IC5Notice: We have upgraded developerWorks Community to the latest version of IBM Connections. For more information, read our upgrade FAQ.
Topic
  • 3 replies
  • Latest Post - ‏2013-03-27T10:40:09Z by SystemAdmin
SystemAdmin
SystemAdmin
4779 Posts

Pinned topic Poor copyback performace with V3700

‏2013-03-11T06:21:09Z |
I’ve got a weird issue that I’ve been unable to resolve. I have a X3650M2 running 2008 Serer R2. This is my management/backup server. I have an iSCSI connection to my new V3700. When I copy files to the SAN it copies very quickly (I can copy 6.2GB in about a minute with speeds of around 128 MB/sec). However, when I copy the same files from the SAN to the management server it takes forever (34 mins with transfer speeds as low as 2.96Mb/sec). It looks like it will start at a high rate but shortly after slow and get progressively slower throughout the operation. I have tested with another box running 2008R2 with an iSCSI connection to the V3700 and it works ok. I also have a DS3300 and have no issues transferring data between the management server and the DS3300. Takes about a minute to move 6.Gb and 4 minutes to copy it back to the management server.

So everything to me points to a particular interaction between the V3700 and the management server. If it was a OS or NIC thing I’d expect similar performance with the DS3300.

I’ve verified all my iSCSI settings. I am not using any multipath, and have also added/removed CHAP. I do not use jumbo frames. I’ve actually got a job logged with IBM but have not resolved the issue.

If anyone has had any similar experience any ideas greatly appreciated.
Updated on 2013-03-27T10:40:09Z at 2013-03-27T10:40:09Z by SystemAdmin
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: Poor copyback performace with V3700

    ‏2013-03-27T07:02:01Z  
    Hi,
    We have had a similar issue using VMware 5.1 write performance to the SAN was 100-130MB/s using single gigabit connection. Reading from the SAN 5-10MB/s. Same results when using an IBM X3550 M4 or IBM X3200 M3 host with the IBM version of VMware 5.1 or vanilla version of VMware 5.1.
  • grouts
    grouts
    44 Posts

    Re: Poor copyback performace with V3700

    ‏2013-03-27T09:21:05Z  
    This sounds like the V3700 is waiting for TCP ACK packets, and the "delayed ACK" setup from the host is causing the V3700 to wait more than it should. This wait induced by the "delayed ACK" will only affect read operations and not write.
    Basically the V3700 does not support "delayed ACK" which is often enabled by default on many host OS's.
    So you need to disable "delayed ACK" on your hosts.
    For Windows Server...
    The settings can be checked and modified using following registry settings. Please use regedit under Subkey: *HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces{Interface GUID}**

    {Interface GUID} is different for every IP, the GUID under which host's iSCSI ip can be seen is the one which requires setting.

    set TcpNoDelay to 1.
    This disables Nagles algorithm.
    This value may not be present by default in your registry.
    You may have to include a new value of type DWORD-32 (REG_DWORD) to do this, then set the value to 1.

    set TcpAckFrequency to 1.
    TCP/IP can be the source of some significant remote method delays.
    Changing this value can increase TCP performance by immediately acknowledging incoming TCP segments.
    This value may not be present by default in your registry.
    You may have to include a new value of type DWORD-32 (REG_DWORD) to do this, then set the value to 1.
    If there is confusion in finding the right iSCSI interface for either of the above values
    (this is usually done by looking at the IPaddress key for each interface),
    the settings can be enabled on each of the interfaces if needed.

    It is important that the Windows host must be rebooted for these settings to take effect.

    For VMware
    Follow VMware KB article to disable delayed ACK (exact instructions vary by ESX level- see KB article for more information on each version).

    Link: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

    Please note: When switching off Disabled Ack, you MUST reboot the ESX server for this to take effect.
  • SystemAdmin
    SystemAdmin
    4779 Posts

    Re: Poor copyback performace with V3700

    ‏2013-03-27T10:40:09Z  
    • grouts
    • ‏2013-03-27T09:21:05Z
    This sounds like the V3700 is waiting for TCP ACK packets, and the "delayed ACK" setup from the host is causing the V3700 to wait more than it should. This wait induced by the "delayed ACK" will only affect read operations and not write.
    Basically the V3700 does not support "delayed ACK" which is often enabled by default on many host OS's.
    So you need to disable "delayed ACK" on your hosts.
    For Windows Server...
    The settings can be checked and modified using following registry settings. Please use regedit under Subkey: *HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces{Interface GUID}**

    {Interface GUID} is different for every IP, the GUID under which host's iSCSI ip can be seen is the one which requires setting.

    set TcpNoDelay to 1.
    This disables Nagles algorithm.
    This value may not be present by default in your registry.
    You may have to include a new value of type DWORD-32 (REG_DWORD) to do this, then set the value to 1.

    set TcpAckFrequency to 1.
    TCP/IP can be the source of some significant remote method delays.
    Changing this value can increase TCP performance by immediately acknowledging incoming TCP segments.
    This value may not be present by default in your registry.
    You may have to include a new value of type DWORD-32 (REG_DWORD) to do this, then set the value to 1.
    If there is confusion in finding the right iSCSI interface for either of the above values
    (this is usually done by looking at the IPaddress key for each interface),
    the settings can be enabled on each of the interfaces if needed.

    It is important that the Windows host must be rebooted for these settings to take effect.

    For VMware
    Follow VMware KB article to disable delayed ACK (exact instructions vary by ESX level- see KB article for more information on each version).

    Link: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1002598

    Please note: When switching off Disabled Ack, you MUST reboot the ESX server for this to take effect.
    I've heard about this setting. The references I saw pertained to clustered Windows servers. I will certainly try this and let you know the outcome.

    Thanks