IT42569: START OF SERVER HANGS AT 100 % CPU LOAD AND FAILS WITH ANR0162W / ANR1820E ON SYSTEMS WITH AMD EPYC PROCESSORS

APAR status

• Closed as suggestion for future release.

Error description

• On systems with AMD EPYC processors, the IBM Spectrum Protect
  server will hang at 100% CPU load after server start and will
  finally fail with ANR0162W and ANR1820E messages.
  
  Here an example for such an event with the IBM Spectrum Protect
  server being started in foreground:
  
  ANR0162W Supplemental database diagnostic information:
  -1:08001:-30081 ([IBM][CLI Driver] SQL30081N  A communication
  error has been detected. Communication protocol being used:
  "TCP/IP".  Communication API being used: "SOCKETS".  Location
  where the error was detected: "127.0.0.1".  Communication
  function detecting the error: "recv".  Protocol specific error
  code(s): "*", "*", "0".  SQLSTATE=08001
  ANR1820E The server could not connect to the database manager by
  using TCP/IP port 51500.~
  
  A server trace while the startup with traceflags  INIT DBCONN
  DBADM DBCLI DBDBG DBRC TCPIP:
  15:30:02.530 [1][dbiconn.c][1914][RdbCreateConnection]:TCP
  connection requested, for >instance user<.
  15:30:02.530 [1][dbiconn.c][1927][RdbCreateConnection]:Using
  connect string
  DATABASE=TSMDB1;PROTOCOL=TCPIP;HOSTNAME=localhost;PORT=51500;UI
  D=>instanceuser<;PWD=****;
  ==> time gap
  17:32:15.045 [1][dbiconn.c][1936][RdbCreateConnection]:TCP
  connection return code -1.
  17:32:15.045
  [1][dbieval.c][667][DbiEvalSQLOutcomeX]:dbiconn.c(1995)
  requesting stmtEval with sqlRc -1 for handle 0:1.
  17:32:15.046 [1][dbieval.c][715][DbiEvalSQLOutcomeX]:SQLSTATE
  08001, SQLCODE -30081, SQLMsg=[IBM][CLI Driver] SQL30081N  A
  communication error has been detected. Communication protocol
  being used: "TCP/IP".  Communication API being used: "SOCKETS".
  Location where the error was detected: "127.0.0.1".
  Communication function detecting the error: "recv".  Protocol
  specific error code(s): "*", "*", "0".  SQLSTATE=08001
  , msgLen=328.
  17:32:15.046 [1][dbiconn.c][1995][TraceMsg]:ANR0162W
  Supplemental database diagnostic information:  -1:08001:-30081
  ([IBM][CLI Driver] SQL30081N  A communication error has been
  detected. Communication protocol being used: "TCP/IP".
  Communication API being used: "SOCKETS".  Location where the
  error was detected: "127.0.0.1".  Communication function
  detecting the error: "recv".  Protocol specific error code(s):
  "*", "*", "0".  SQLSTATE=08001
  ).~
  17:32:15.046 [1][dbieval.c][806][DbiEvalSQLOutcomeX]:Set retRc
  2339 on sqlRc -1 for handleType 2.
  17:32:15.046 [1][dbiconn.c][1999][RdbCreateConnection]:Error -1
  connecting to database TSMDB1.
  17:32:15.046 [1][dbiconn.c][673][DbiGetConnectionTracked]:Exit,
  rc, 9994, connP (nil).
  17:32:15.046 [1][rdbinst.c][2231][RdbTestConnectionToDB2X]:Error
  9994 getting connection to db2 using type=2.
  17:32:15.046 [1][rdbinst.c][2306][RdbTestConnectionToDB2X]:Exit
  rc 4521.
  17:32:15.046 [1][dbiinit.c][1093][TraceMsg]:ANR1820E The server
  could not connect to the database manager by using TCP/IP port
  51500.~
  
  dsmdb2pw trace shows:
  db2secServerAuthPluginInit[1434] No DB2_ICC_IGNORE_FIPS variable
  found.
  initICC[1499] Libpath for ICC is /usr/local/ibm/gsk8_64/lib64
  ValidatePassword[1090] Digest doesn't match cache - reading
  file.
  ReadSha1[508] Opening file /home/tsm02/sqllib/tsmdbdata.TSMDB1
  to read digest
  ValidatePassword[1097] Digest doesn't match new value - failing
  request.
  ValidatePasswordWithOS[1891] Connecting to auth server at
  address >instance directory</sqllib/tsmdbauth
  ValidatePasswordWithOS[1900] Error 111 connecting to server.
  sun_family is 1, sun_path=" address >instance
  directory</sqllib/tsmdbauth" length=30
  
  where Errno 111 is ECONNREFUSED Connection refused.
  
  A connection to the db2 also will fail:
  db2start
  db2 connect to tsmdb1 USER >instance user<
  Enter current password for >instance user<:
  
  SQL30082N Security processing failed with reason "15"
  ("PROCESSING FAILURE").
  SQLSTATE=08001
  
  IBM Spectrum Protect versions Affected:
  
  All IBM Spectrum Protect Server versions on all supported
  platforms with AMD EPYC processors
  
  Additional Keywords: TS011041413 dsmserv hang CPU AMD EPYC
  

Local fix

• If the IBM Spectrum Protect server delivers GSkit build
  8.0.50.86 or above, the following workaround can be applied:
  
  1) Issue "export ICC_SHIFT=3" in the environment where the
     application is running.
  
  Or:
  
  2) Look for the ICCSIG.txt files in the  C/icc/icclib and
     N/icc/icclib  directories.
  
     Append  ICC_SHIFT=3  to the ICCSIG.txt files.
  

Problem summary

Problem conclusion

Temporary fix

Comments

• This APAR is being cancelled because there exists a workaround
  which the customers can use for affected processor type.
  

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels