IT01913: AFTER TPC-R UPGRADE TO 5.2, TPC-R IS UNABLE TO CONNECT TO DS8K.

Fix Pack 5.2.4 (November 2014) for Tivoli Storage Productivity Center
Fix Pack 5.2.4.1 (December 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.5 (March 2015) for Tivoli Storage Productivity Center (withdrawn)
Fix Pack 5.2.5.1 (April 2015) for Tivoli Storage Productivity Center (withdrawn)
Refresh Pack 5.2.6 (June 2015) for Tivoli Storage Productivity Center
Refresh Pack 5.2.3 (August 2014) for Tivoli Storage Productivity Center
Refresh Pack 5.2.7 (August 2015) for Tivoli Storage Productivity Center
IBM Spectrum Control V5.2.8 (December 2015)
IBM Spectrum Control V5.2.9 (February 2016)
IBM Spectrum Control V5.2.10 (May 2016)
IBM Spectrum Control V5.2.10.1 (July 2016)
IBM Spectrum Control V5.2.11 (August 2016)
IBM Spectrum Control V5.2.12 (November 2016)
IBM Spectrum Control V5.2.13 (March 2017)
IBM Spectrum Control V5.2.14 (May 2017)
IBM Spectrum Control V5.2.15 (August 2017)
IBM Spectrum Control V5.2.15.2 (November 2017)
IBM Spectrum Control V5.2.16 (March 2018)
IBM Spectrum Control V5.2.17 (May 2018)
Fix Pack 5.2.7.1 (February 2016) for Tivoli Storage Productivity Center

APAR status

• Closed as fixed if next.

Error description

• TPC-R 5.2 ESSNI client will first try to use port 1751 which
  supports NIST SP 800-131a-compliant certificates.
  If the DS8K microcode does not support port 1751,  TPC-R tries
  port 1750.
  In the essni client code, they did not specify a timeout value
  for the request on port 1751 so it defaults to the host timeout
  value which is too long.
  By the time the ESSNI client tries the other port 1750, the
  command times out.
  
  In TPC-R trace logs,  may see the following message.
  DEVICES HAVE LOST CONNECTION!!!
  

Local fix

• Apply fixpack or contact IBM support (HW/DS8K team) for
  workaround.
  

Problem summary

• | fix pack | 5.2.3-TIV-TPC-FP0000 - target 3Q 2014 |
  
  
  http://www-01.ibm.com/support/docview.wss?&uid=swg21320822
  
  The target dates for future fix packs do not represent a formal
  commitment by IBM. The dates are subject to change without
  notice.
  
  The interface to the hardware layer has a bug that will cause a
  memory leak in any pre R7 Device.
  Problem:
  Starting with DS8k R7.2, NI client is using new port 1751 to
  connect to server, if the connection fails, it will use old 1750
  port. Unfortunately, NI client doesn't specify a timeout value
  when opening socket on port 1751, so it depends on the OS to
  supply the default timeout value. When DS8k microcode lower than
  R7.2 is in use, which doesn't support port 1751, client opening
  socket to server will fail but it takes too long time on some
  OS. In the worst case, it also hangs at opening the socket. As a
  result, NI client cannot connect to DS8K. Since NI client is the
  interface that TPC-R uses for an HMC connection TPC-R will also
  display this as a connection error. The NI Client may start
  consuming large amount of memory as well eventually leading to
  an out of memory error.
  

Problem conclusion

• A new product dependency essni client jar was shipped with the
  product to pick up this fix.
  

Temporary fix

• Workaround 1 on HMC server side:
  1. Run iptables commands on HMC server to allow client to
  connect from all kinds of operating systems.
    iptables --append INPUT -i eth2 -p tcp --dport 1751 -j ACCEPT
    ip6tables --append INPUT -i eth2 -p tcp --dport 1751 -j ACCEPT
  
  2. The configuration might be clear after HMC reboot. To allow
  it persistent across HMC reboots, need to update file
  /opt/esshmc/data/firewall on HMC and added following rules:
    $iptables --append INPUT -i eth2 -p tcp --dport 1751 -j ACCEPT
  &&
    $ip6tables --append INPUT -i eth2 -p tcp --dport 1751 -j
  ACCEPT &&
  
  3.  If the DS8k has a Microcode upgrade before application picks
  up NI client fix, the firewall setting for port 1751 will need
  to be updated again.
  
  Workaround 2 on application side:
        In case customer don't want to do any change on HMC side,
  we have different workaround on application side:
  
  AIX:
  1. Issue "/usr/sbin/no -a" on AIX machine, find tcp_keepinit.
  for example, tcp_keepinit=150
  2. Change this value to a smaller one. using " /usr/sbin/no -o
  tcp_keepinit=50"
  3. Restart application.
  
  Linux:
  run command: sysctl -w net.ipv4.tcp_syn_retries=2
  
  z/OS:
   TCP configuration cannot be changed on z/OS so it has to change
  the iptables on HMC server using workaround 1.
  
  Windows:
  The default timeout value on Windows is small, we haven't see
  this issue on Windows.
  

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels

• R522 PSY

     UP

• R510 PSN

     UP

• R511 PSN

     UP

• R520 PSN

     UP

• R521 PSN

     UP