TCP streaming workload tuning
Streaming workloads move large amounts of data from one endpoint to the other endpoint. Examples of streaming workloads are file transfer, backup or restore workloads, or bulk data transfer. The main metric of interest in these workloads is bandwidth, but you can also look at end-to-end latency.
The primary tunables that affect TCP performance for streaming applications are the following:
- tcp_recvspace
- tcp_sendspace
- rfc1323
- MTU path discovery
- tcp_nodelayack
- sb_max
- Adapter options, such as checksum offload and TCP Large Send
Device | Speed | MTU size | tcp_sendspace | tcp_recvspace | sb_max 1 | rfc1323 |
---|---|---|---|---|---|---|
Token Ring | 4 or 16 Mbit | 1492 | 16384 | 16384 | 32768 | 0 |
Ethernet | 10 Mbit | 1500 | 16384 | 16384 | 32768 | 0 |
Ethernet | 100 Mbit | 1500 | 16384 | 16384 | 65536 | 0 |
Ethernet | Gigabit | 1500 | 131072 | 65536 | 131072 | 0 |
Ethernet | Gigabit | 9000 | 131072 | 65535 | 262144 | 0 |
Ethernet | Gigabit | 9000 | 262144 | 131072 2 | 524288 | 1 |
Ethernet | 10 Gigabit | 1500 | 131072 | 65536 | 131072 | 0 |
Ethernet | 10 Gigabit | 9000 | 262144 | 131072 | 262144 | 1 |
ATM | 155 Mbit | 1500 | 16384 | 16384 | 131072 | 0 |
ATM | 155 Mbit | 9180 | 65535 | 65535 3 | 131072 | 0 |
ATM | 155 Mbit | 65527 | 655360 | 655360 4 | 1310720 | 1 |
FDDI | 100 Mbit | 4352 | 45056 | 45056 | 90012 | 0 |
Fibre Channel | 2 Gigabit | 65280 | 655360 | 655360 | 1310720 | 1 |
(1) It is suggested to use the default value of 1048576 for the sb_max tunable. The values shown in the table are acceptable minimum values for the sb_max tunable.
(2) Performance is slightly better when using these options, with rfc1323 enabled, on jumbo frames on Gigabit Ethernet.
(3) Certain combinations of TCP send and receive space will result in very low throughput, (1 Mbit or less). To avoid this problem, set the tcp_sendspace tunable to a minimum of three times the MTU size or greater or equal to the receiver's tcp_recvspace value.
- Enable rfc1323, which enhances TCP and allows it to overcome the 16-bit limit so that it can use a window size larger than 64 KB. You can then set the tcp_recvspace tunable to a large value, such as 10 times the MTU size, which allows TCP to stream data and thus provides good performance.
- Reduce the MTU size of the adapter. For example, use the ifconfig at0 mtu 16384 command to set the ATM MTU size to 16 KB. This causes TCP to compute a smaller MSS value. With a 16 KB MTU size, TCP can send four packets for a 64 KB window size.
The following are general guidelines for tuning TCP streaming workloads:
- Set the TCP send and receive space to at least 10 times the MTU size.
- You should enable rfc1323 when MTU sizes are above 8 KB to allow larger TCP receive space values.
- For high speed adapters, larger TCP send and receive space values help performance.
- For high speed adapters, the tcp_sendspace tunable value should be 2 times the value of tcp_recvspace.
- The rfc1323 for the lo0 interface is set by default. The default MTU size for lo0 is higher than 1500, so the tcp_sendspace and tcp_recvspace tunables are set to 128K.
The ftp and rcp commands are examples of TCP applications that benefit from tuning the tcp_sendspace and tcp_recvspace tunables.