iSCSI performance analysis and tuning

This procedure provides a solution for Internet Small Computer Systems Interface (iSCSI) host performance problems while connected to a system and its connectivity to the network switch.

1. Disable the TCP delayed acknowledgment feature.
   To disable this feature, refer to OS/platform documentation.

   • VMware: https://kb.vmware.com/s/article/1002598.
   • Windows: http://support.microsoft.com/kb/823764.
   • Linux (RHEL): https://access.redhat.com/solutions/407743.
   • Linux (Other): Use IP utility to enable quickack on host side. e.g ip route show; ip route change 192.168.140.0/24 dev eth7 proto kernel scope link src 192.168.140.55 quickack 1.

   The primary signature of this issue: read performance is significantly lower than write performance. Transmission Control Protocol (TCP) delayed acknowledgment is a technique that is used by some implementations of the TCP to improve network performance. However, in this scenario where the number of outstanding I/O is 1, the technique can significantly reduce I/O performance.

   In essence, several ACK responses can be combined into a single response, reducing protocol overhead. As described in RFC 1122, a host can delay sending an ACK response by up to 500 ms. Additionally, with a stream of full-sized incoming segments, ACK responses must be sent for every second segment.

2. Enable jumbo frame for iSCSI.

   Jumbo frames are Ethernet frames with a size in excess of 1500 bytes. The maximum transmission unit (MTU) parameter is used to measure the size of jumbo frames.

   The system supports 9000-bytes MTU. Refer to the chportethernet command to enable jumbo frame. This command is disruptive as the link flips and the I/O operation through that port pauses.

   The network must support jumbo frames end-to-end to be effective. Send a ping packet to be delivered without fragmentation to verify that the network supports jumbo frames. For example:

   • Windows:

     ping -t <iscsi target ip> -S <iscsi initiator ip> -f -l <new mtu size - packet overhead (usually 36, might differ)>

     The following command is an example of a command that is used to check whether a 9000-bytes MTU is set correctly on a Windows 7 system:

     ping -t -S 192.168.1.117 192.168.1.217 -f -l 8964

     The following output is an example of a successful reply:

     192.168.1.217: bytes=8964 time=1ms TTL=52

   • Linux®:

     ping -l <source iscsi initatior ip> -s <new mtu size> -M do <iscsi target ip> 

   • ESXi:

     ping <iscsi target ip> -I <source iscsi initiator ip> -s <new mtu size - 28> -d

3. Verify the switch's port statistic where initiator/target ports are connected to make sure that packet drops are not high.

   Review network architecture to avoid any bottlenecks and oversubscription. The network needs to be balanced to avoid any packet drop; packet drop significantly reduces storage performance. Involve networking support to fix any such issues.

4. Optimize and utilize all iSCSI ports.
   To optimize system resource utilization, all iSCSI ports must be used.

   • Each port is assigned to one CPU, and by balancing the login, one can maximize CPU utilization and achieve better performance. Ideally, configure subnets equal to the number of iSCSI ports on the system node. Configure each port of a node with an IP on a different subnet and keep it the same for other nodes. The following example displays an ideal configuration:
     
Node 1
Port 1: 192.168.1.11
Port 2: 192.168.2.21
Port 3: 192.168.3.31

Node 2:
Port 1: 192.168.1.12
Port 2: 192.168.2.22
Port 3: 192.168.3.33

   • Avoid situations where 50 hosts are logged in to port 1 and only five hosts are logged in to port 2.
   • Use proper subnetting to achieve a balance between the number of sessions and redundancy.
5. Ensure that proper bandwidth is given to iSCSI on the network.

   You can divide the bandwidth among the various types of traffic. It is important to assign proper bandwidth for good performance. To assign bandwidth for iSCSI traffic, you need to first enable the priority flow control for iSCSI.