IBM Support

Slow performance and frequent disconnection from an iSCSI LUN when used with FastBack Mount

Troubleshooting


Problem

Read operations process slowly or are disconnected when an iSCSI LUN is used with FastBack Mount.

Symptom

FastBack Mount is used to mount a protected Linux volume snapshot to an Internet Small Computer System Interface (iSCSI) logical unit number (LUN). This LUN is then made available to a Linux system. The Linux system uses Open-iSCSI to create and read a device for this LUN.

This FastBack Mount feature is frequently used with Linux file-level restore and the Tivoli Storage Manager Backup-Archive Client BACKUP FASTBACK command. This command backs up Linux volume snapshots to a Tivoli Storage Manager server. When the BACKUP FASTBACK command is issued, FastBack Mount reads data from the iSCSI LUN. In some environments, this read action results in frequent disconnections and operation time outs. As a result, performance becomes slow.

Diagnosing The Problem

To verify the problem, check whether any time out or connection errors (#1011) were logged by Open-iSCSI. This example shows how to check for these errors:

Enter this command at the command prompt:

    cat /var/log/messages | grep 1011

If any time out or connection errors (#1011) were logged by Open-iSCSI, they will display as shown here:
    Jan 22 01:37:15 parkcity kernel: connection2:0: detected conn error (1011)
    Jan 22 01:37:16 parkcity iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
    Jan 22 01:37:33 parkcity kernel: connection2:0: detected conn error (1011)
    Jan 22 01:37:34 parkcity iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)

You can also verify this problem by checking for time out or connection errors (#1011) using the appropriate time stamp. This example shows how to check for these errors using a time stamp
of 01:37:3x.

Enter this command at the command prompt:
    cat /var/log/messages | grep 01:37:3

Time out or connection errors (#1011) that were logged by Open-iSCSI, with a time stamp of 01:37:3x, will display as shown here:
    Jan 22 01:37:33 parkcity kernel: connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx
    4440525921, last ping 4440525921, now 4440535921
    Jan 22 01:37:33 parkcity kernel: connection2:0: detected conn error (1011)
    Jan 22 01:37:33 parkcity kernel: sd 4:0:0:0: SCSI error: return code = 0x000e0000
    Jan 22 01:37:33 parkcity kernel: end_request: I/O error, dev sdu, sector 156790895
    Jan 22 01:37:34 parkcity iscsid: Kernel reported iSCSI connection 2:0 error (1011) state (3)
    Jan 22 01:37:37 parkcity iscsid: connection2:0 is operational after recovery (1 attempts)

You can also verify this problem by checking for time out or connection errors by matching the LUN in the error with the LUN associated with the mounted iSCSI. This example shows how to check by matching LUN 4:0:0:0.

Enter this command at the command prompt:
    -bash-3.2# lsscsi | grep FastBack

Time out or connection errors associated with the LUN (4:0:0:0) will display as shown here:
    [4:0:0:0] disk IBM TSM FastBack 0.1 /dev/sd

Resolving The Problem


Temporary Solution:

If it is not possible to reboot the machine (for example, the machine is a production machine), the temporary solution is to disable TCM Windows Scaling. This temporary solution resolves the issue until a reboot is performed.

    Important: These changes are no longer in effect after the machine is rebooted.

1. Locate the current TCP Window Scaling setting by entering this command:

    cat /proc/sys/net/ipv4/tcp_window_scaling

    If the result is "0", then TCP Window Scaling is disabled.


    If the result is "1", then TCP Window Scaling is enabled. It must be disabled for this temporary solution.

2. If TCP Window Scaling is enabled, disable it by performing these tasks:
      (a) Unmount all FastBack iSCSI LUNs and log out of the iSCSI initiator. If the Tivoli Storage Manager Client BACKUP FASTBACK command is in use, exit the dsmc application.

      (b) Change the TCP Window Scaling setting to 0 by issuing this command:

        echo 0 > /proc/sys/net/ipv4/tcp_window_scaling
    The TCP Window Scaling should be disabled immediately.

3. Remount your FastBack iSCSI LUNs. You can also use the BACKUP FASTBACK command (if needed).


Permanent Solution:
If it is possible to reboot the machine, these tasks provide a permanent solution to this issue.

Change to the /etc directory:
    cd /etc

Enter this command at the command prompt:

    grep tcp_window_scaling sysctl.conf
If nothing prints (for example, the file exists but does not contain the relevant text), issue these commands:
    echo "# Disable TCP window scaling" >> sysctl.conf
    echo "net.ipv4.tcp_window_scaling=0" >> sysctl.conf
If the grep command prints a "No such file or directory" message, issue these commands:
    echo "# Disable TCP window scaling" > sysctl.conf
    echo "net.ipv4.tcp_window_scaling=0" >> sysctl.conf
If the grep command prints "net.ipv4.tcp_window_scaling=0", then TCP window scaling has already been disabled.

[{"Product":{"code":"SS9NU9","label":"Tivoli Storage Manager FastBack"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"--","Platform":[{"code":"PF016","label":"Linux"}],"Version":"6.1","Edition":"All Editions","Line of Business":{"code":"LOB26","label":"Storage"}}]

Document Information

Modified date:
17 June 2018

UID

swg21420774