IT26211: SPECTRUM PROTECT SERVER CAN CRASH DUE TO A DEADLOCK IN DB2.

7.1.10.000-TIV-TSMSRV-WIN
7.1.10.000-TIV-TSMSRV-SolarisSPARC
7.1.10.000-TIV-TSMSRV-Linuxx86_64
7.1.10.000-TIV-TSMSRV-Linuxs390x
7.1.10.000-TIV-TSMSRV-Linuxppc64
7.1.10.000-TIV-TSMSRV-HP-UX
7.1.10.000-TIV-TSMSRV-AIX
7.1.7.500-TIV-TSMSRV-WIN
7.1.7.500-TIV-TSMSRV-SolarisSPARC
7.1.7.500-TIV-TSMSRV-Linuxx86_64
7.1.7.500-TIV-TSMSRV-Linuxs390x
7.1.7.500-TIV-TSMSRV-Linuxppc64
7.1.7.500-TIV-TSMSRV-HP-UX
7.1.7.500-TIV-TSMSRV-AIX
8.1.7.000-IBM-SPSRV-Linuxppc64le
8.1.7.000-IBM-SPCMS-WindowsX64
8.1.7.000-IBM-SPCMS-WindowsI32
8.1.7.000-IBM-SPCMS-Linuxx86_64
8.1.7.000-IBM-SPOC-WindowsX64
8.1.7.000-IBM-SPOC-Linuxx86_64
8.1.7.000-IBM-SPOC-Linuxs390x
8.1.7.000-IBM-SPOC-LinuxPPC64le
8.1.7.000-IBM-SPOC-AIX
8.1.7.000-IBM-SPSRV-WindowsX64
8.1.7.000-IBM-SPSRV-Linuxx86_64
8.1.7.000-IBM-SPSRV-Linuxs390x
8.1.7.000-IBM-SPSRV-AIX
7.1.9.100-TIV-TSMSRV-AIX
7.1.9.100-TIV-TSMSRV-HP-UX
7.1.9.100-TIV-TSMSRV-Linuxppc64
7.1.9.100-TIV-TSMSRV-Linuxs390x
7.1.9.100-TIV-TSMSRV-Linuxx86_64
7.1.9.100-TIV-TSMSRV-SolarisSPARC
7.1.9.100-TIV-TSMSRV-WIN
IBM Spectrum Protect Server V7.1.9.X interim fix downloads
IBM Spectrum Protect Server V8.1.6.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 7 (V8.1.7) Downloads
IBM Spectrum Protect Server V7.1.7.X interim fix downloads
IBM Spectrum Protect Server V7.1 Fix Pack 10 (7.1.10.000) Downloads

APAR status

• Closed as program error.

Error description

• Prior to the server crash, the following messages can be seen
  from activity logs indicating the occurrence of database
  deadlock:
  ANR0159E sdrefcount.c(390): Database deadlock detected on 115:2.
  ANR0162W Supplemental database diagnostic information:
                            -1:40001:-911 ([IBM][CLI
  Driver][DB2/AIX64] SQL0911N  The
                            current transaction has been rolled
  back because of a
                            deadlock or timeout.  Reason code
  "68".  SQLSTATE=40001).
  
  Then after getting the following errors which lead to an
  aborted transaction:
  
  ANR9999D Thread<15396>  0x000000010004785c tmAbortX
  ANR9999D Thread<15396>  0x00000001009cd530 QueueRefCountUpdates
  ANR9999D Thread<15396>  0x00000001009cbd64 SdFlushWriteControl
  
  The Spectrum Protect server can crash with the following call
  stack seen from the dsmserv core file:
  
  tmPrtTid(??, ??) at 0x10004aa14
  tmEndWithStreamMsg(??, ??, ??, ??) at 0x100048454
  SdFlushWriteControl(??) at 0x1009cbeec
  SdCQSinkThread(??) at 0x1009c1ca0
  StartThread(0x0) at 0x10000da90
  
  Notes: The above ANR9999D messages might not have made it to
  the activity log. If they are not seen in the activity log,
  then they are still in the dump and can be found by examining
  the strings in the dump, for example:
  On AIX system:
  strings name_of_core | grep ANR9999D
  IBM Spectrum Protect versions affected:
  Spectrum Protect Server version 7.1.3 and above on all
  supported platforms.
  
  Customer/L2 Diagnostics (If Applicable):
  The following messages can be seen from activity log prior to
  the server crash:
  ANR0162W Supplemental database diagnostic information:
                            -1:40001:-911 ([IBM][CLI
  Driver][DB2/AIX64] SQL0911N  The
                            current transaction has been rolled
  back because of a
                            deadlock or timeout.  Reason code
  "68".  SQLSTATE=40001).
  ANR9999D_0986765464 tmAbortX(tmtxn.c:1297) Thread<15397>:
                            Invalid transaction descriptor: 0
  ANR9999D Thread<15397> issued message 9999 from:
  ANR9999D Thread<15397>  0x000000010001fd94 StdPutText
  ANR9999D Thread<15397>  0x00000001000209b8 OutDiagToCons
  ANR9999D Thread<15397>  0x000000010000a6f4 outDiagfExt
  ANR9999D Thread<15397>  0x000000010004785c tmAbortX
  ANR9999D Thread<15397>  0x00000001009cd530
                            QueueRefCountUpdates
  ANR9999D Thread<15397>  0x00000001009cbd64
                            SdFlushWriteControl
  ANR9999D Thread<15397>  0x00000001009c1ca4 SdCQSinkThread
  ANR9999D Thread<15397>  0x000000010000da94 StartThread
  Initial Impact:
  Medium
  
  Additional Keywords:
  TS001302059, Spectrum Protect Server, ANR9999D, deadlock, TSM
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 7.1.9.100, 7.1.10,           *
  * 8.1.6.100, and 8.1.7. Note that this is subject to change at *
  * the discretion of IBM.                                       *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Solaris, Linux,
  and Windows.
  Platforms fixed:  AIX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R71A PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP