4285ch04.fm Draft Document for Review May 4, 2007 11:35 am
128 Linux Performance and Tuning Guidelines
125Mbytes/sec (1Gbit/sec) * 1msec = 125Kbytes
As the default value of rmem_max and wmem_max are about 128Kbytes in most enterprise
distributions, it may be fair enough for low-latency general purpose network environment.
However if the latency is large, the default size may be too small.
Taking another example, assuming that a samba file server has to support 16 concurrent file
transfer session from various locations, socket buffer size for each session comes down to
8Kbytes in default configuration. This may be relatively small if the data transfer is high.
򐂰Set the max OS send buffer size (wmem) and receive buffer size (rmem) to 8 MB for
queues on all protocols:
sysctl -w net.core.wmem_max=8388608
sysctl -w net.core.rmem_max=8388608
These specify the amount of memory that is allocated for each TCP socket when it is
created.
򐂰In addition, you should also use the following commands for send and receive buffers.
They specify three values: minimum size, initial size, and maximum size:
sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608"
sysctl -w net.ipv4.tcp_wmem="4096 87380 8388608"
The third value must be the same as or less than the value of wmem_max and
rmem_max. However we also suggest increasing the first value on high-speed,
high-quality networks so that the TCP windows start out at a sufficiently high value.
򐂰Increase the values in /proc/sys/net/ipv4/tcp_mem. The three values refer to minimum,
pressure, and maximum memory allocations for TCP memory
You can see what’s been changed by socket buffer tuning using tcpdump. As the examples
show, limiting socket buffer to small size results in small window size and causes frequent
acknowledgement packets and inefficient use (Example4-19). On the contrary, making
socket buffer large results in a large window size (Example4-20).
Example 4-19 Small window size (rmem, wmem=4096)
[root@lnxsu5 ~]# tcpdump -ni eth1
22:00:37.221393 IP plnxsu4.34087 > plnxsu5.32837: P 18628285:18629745(1460) ack 9088 win 46
22:00:37.221396 IP plnxsu4.34087 > plnxsu5.32837: . 18629745:18631205(1460) ack 9088 win 46
22:00:37.221499 IP plnxsu5.32837 > plnxsu4.34087: . ack 18629745 win 37
22:00:37.221507 IP plnxsu4.34087 > plnxsu5.32837: P 18631205:18632665(1460) ack 9088 win 46
22:00:37.221511 IP plnxsu4.34087 > plnxsu5.32837: . 18632665:18634125(1460) ack 9088 win 46
22:00:37.221614 IP plnxsu5.32837 > plnxsu4.34087: . ack 18632665 win 37
22:00:37.221622 IP plnxsu4.34087 > plnxsu5.32837: P 18634125:18635585(1460) ack 9088 win 46
22:00:37.221625 IP plnxsu4.34087 > plnxsu5.32837: . 18635585:18637045(1460) ack 9088 win 46
22:00:37.221730 IP plnxsu5.32837 > plnxsu4.34087: . ack 18635585 win 37
22:00:37.221738 IP plnxsu4.34087 > plnxsu5.32837: P 18637045:18638505(1460) ack 9088 win 46
22:00:37.221741 IP plnxsu4.34087 > plnxsu5.32837: . 18638505:18639965(1460) ack 9088 win 46
22:00:37.221847 IP plnxsu5.32837 > plnxsu4.34087: . ack 18638505 win 37
Example 4-20 Large window size (rmem, wmem=524288)
[root@lnxsu5 ~]# tcpdump -ni eth1
22:01:25.515545 IP plnxsu4.34088 > plnxsu5.40500: . 136675977:136677437(1460) ack 66752 win 46
22:01:25.515557 IP plnxsu4.34088 > plnxsu5.40500: . 136687657:136689117(1460) ack 66752 win 46
22:01:25.515568 IP plnxsu4.34088 > plnxsu5.40500: . 136699337:136700797(1460) ack 66752 win 46
22:01:25.515579 IP plnxsu4.34088 > plnxsu5.40500: . 136711017:136712477(1460) ack 66752 win 46
22:01:25.515592 IP plnxsu4.34088 > plnxsu5.40500: . 136722697:136724157(1460) ack 66752 win 46