IT26635: SERVER CRASH CAN EXHIBIT ONE OR MORE SYMPTOMS

APAR status

Closed as program error.

Error description

A Spectrum Protect server crash can exhibit one or more of the
following symptoms:

If the ANR1880W message is found in the activity log, Db2 will
likely be restarted and unpredicable behavior may occur
including certain ANR9999D messages followed by a crash of the
Spectrum Protect server address space.
  ANR1880W Server transaction was canceled because of a
conflicting lock on table table name.

    Note:  If the ANR1880W is encountered then you may be
hitting APAR "IT25877: INDEX REORG IN LOCKWAIT CAN CANCEL LONG
RUNNING TRANSACTION",  The local fix recommends to disable index
reorg.

Customer/L2 Diagnostics:

Certain ANR9999D messages may not have been written to the
activity log prior to a crash and so the following method has
been effective on AIX core dumps to identify symptoms related to
this APAR.
   strings <core_file_name> | grep 9999D

Examine output and look for various ANR9999D call stacks such
as:

   ANR9999D Thread<172760>  0x000000010012ec80
IPRA.$HandleShortCircuitCodes ~(PROCESS: 2750)
   ANR9999D Thread<172760>  0x000000010012e1ac
DbiEvalSQLOutcomeX ~(PROCESS: 2750)
   ANR9999D Thread<172760>  0x0000000100103e84 TblClose
~(PROCESS: 2750)
   ANR9999D Thread<172760>  0x000000010019b3ac IPRA.$FreeTxnDesc
~(PROCESS: 2750)
   ANR9999D Thread<172760>  0x000000010019b088 dbiEndTxn
~(PROCESS: 2750)

or specific ANR9999D message such as:

    ANR9999D_3095886799 HandleShortCircuitCodes(dbieval.c:1166)
Thread<62>: Invalid handle used from dbitxn.c(787).


If the Spectrum Protect address space crashes, the following
call stacks may also indicate the problem has occurred.

   abort.abort() at 0x90000000005e538
   omrsignal.masterSynchSignalHandler() at 0x90000000e5edb78
   dbitxn.dbiPrepareTxn(??, ??, ??) at 0x10019aa1c
   tmtxn.IPRA.$CollectVotes(??) at 0x100054b30
   tmtxn.tmEndX(??, ??, ??) at 0x100053fa8
   smnode.SmEndVbTxn(??, ??, ??, ??, ??, ??) at 0x1006abf10
   smnode.SmNodeSession(??, ??) at 0x1006a06dc
   smsched.SmSchedSession(??) at 0x100671cdc
   smexec.IPRA.$HandleNodeSession(??, ??, ??) at 0x10068e9fc
   smexec.smExecuteSession(??, ??, ??, ??, ??, ??, ??, ??) at
0x100679f9c
   tcpcomm.psSessionThread(??) at 0x100346c90

or

   outDiagfExt(0x100004ca8, 0x54, 0x1212a1050, 0x101411384,
0x111742f98, 0x6f7f6963, 0x6f5f6160,
   0x64626963) at 0x10000b380
   RdbCheckConn(??) at 0x10012d550
   IPRA.$RemoveFromInuseList(??, ??) at 0x10012c5fc
   DbiReleaseConnectionEx(0x111742f98, 0x0, 0x0,
0xfffffffffffffffe) at 0x10012c3a0
   dbitxn.IPRA.$FreeTxnDesc(0x111743498, 0x0, 0x0) at
0x10019b414
   dbiEndTxn(??, ??, ??) at 0x10019b084
   IPRA.$DoEndFuncCallbacks(??, ??) at 0x100055620
   tmAbortX(??, ??, ??) at 0x100053ae8
   bfDerefDeleteThread(??) at 0x100b48fd8

or

   pthread_kill(??, ??) at 0x9000000005678a4
   _p_raise(??) at 0x9000000005670e4
   raise.raise(??) at 0x90000000003fb68
   abort() at 0x90000000005e538
   masterSynchSignalHandler() at 0x90000002aab4b78
   truncate__12GSKASNBufferFUi(??, ??) at 0x9000000030bbbbc
   Encrypt__10KRYContextFRC13GSKASNCBuffer(0x0, 0x16f2a5ba0,
0x0) at 0x9000000028a58c4

SSL_WriteCompressedFragment__13SSLV3ProtocolFRC13GSKASNCBufferUc
(??, ??, ??) at 0x9000000028cdc6c
   SSL_WriteFragment__13SSLV3ProtocolFRC13GSKASNCBufferUc(??,
??, ??) at 0x9000000028cc91c
   SSL_Write__13SSLV3ProtocolFPCviUc(??, ??, ??, ??) at
0x9000000028cb904
   Send__13SSLV3ProtocolFPCvi(??, ??, ??) at 0x9000000028fa74c
   Send__18SSLProtocolManagerFPCvi(??, ??, ??) at
0x9000000028f7eb0
   gsk_secure_soc_write(??, ??, ??, ??) at 0x900000002624790
   tlsSend(??, ??, ??) at 0x1003426ac
   smFlushBufferData(??) at 0x10030416c
   smRecvBufferData(??, ??, ??) at 0x1003168f4
   smRecvBufData@AF63_58(??, ??, ??, ??, ??, ??) at 0x10031786c
   S_Read__FP9SSLHandlePvi(??, ??, ??) at 0x9000000028889a4
   GetV3HeaderInternal__13SSLV3ProtocolFv(??) at
0x9000000028c0058
   GetV3Header__13SSLV3ProtocolFv(??) at 0x9000000028c0c20
   Receive__13SSLV3ProtocolFPvi(??, ??, ??) at 0x9000000028fbc24
   Receive__18SSLProtocolManagerFPvi(??, ??, ??) at
0x9000000028f7e30
   gsk_secure_soc_read(??, ??, ??, ??) at 0x90000000262263c
   tlsRecv(??, ??, ??) at 0x100342a64
   IPRA.$ReceiveVerb(??, ??) at 0x100301bb4
   SmRecvVerbX(??) at 0x1002fb4f4

Additional conditions leading to the problem may be found in the
DB2 diag log showing a NUM_LOG_SPAN Error condition that may
cause aforementioned ANR9999D's and a possible crash of the
Spectrum Protect server address space.

NUM_LOG_SPAN Error event in the db2diag.log for the tsminst1
instance such as:

2015-02-06-07.00.16.197511+060 E60627519A710      LEVEL: Error
PID     : 9875                 TID  : 4397820012816PROC :
db2sysc 0
INSTANCE: tsminst1             NODE : 000         DB   : TSMDB1
EDUID   : 65                   EDUNAME: db2loggr (TSMDB1) 0
FUNCTION: DB2 UDB, data protection services,
sqlpScanTranTableForLowTran, probe:550
MESSAGE : ADM1541W  Application "dsmserv" with application
handle "0-60170" and
          application id "*LOCAL.tsminst1.150206003551"
executing under
          authentication id "TSMINST1" has been forced off of
the database for
          violating database configuration parameter
NUM_LOG_SPAN (current
          value "29"). The unit of work will be rolled back.


*NOTE*
The server may remain up and running after DB2 is restarted
following any one of the error conditions outlined above:

Platforms affected:
Linux/Unix Windows 7.1 and 8.1

Initial Impact: Medium

Additional Keywords: crash core abend TS001403287 TS001434337
TS001434345

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:                                              *
* All IBM Spectrum Protect server users.                       *
****************************************************************
* PROBLEM DESCRIPTION:                                         *
* See error description.                                       *
****************************************************************
* RECOMMENDATION:                                              *
* Apply fixing level when available. This problem is currently *
* projected to be fixed in levels 7.1.9.200, 7.1.10, 8.1.6.100 *
* and 8.1.7. Note that this is subject to change at the        *
* discretion of IBM.                                           *
****************************************************************

Problem conclusion

```
This problem was fixed.
```

Temporary fix

Comments

APAR Information

APAR number
IT26635
Reported component name
TSM SERVER
Reported component ID
5698ISMSV
Reported release
81A
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2018-10-16
Closed date
2018-12-04
Last modified date
2019-01-21

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
TSM SERVER
Fixed component ID
5698ISMSV

Applicable component levels

R71A PSY
UP
R71H PSY
UP
R71L PSY
UP
R71S PSY
UP
R71W PSY
UP
R81A PSY
UP
R81L PSY
UP
R81W PSY
UP

[{"Line of Business":{"code":"LOB26","label":"Storage"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"81A"}]

Document Information

Modified date:
09 September 2021

Tips

IT26635: SERVER CRASH CAN EXHIBIT ONE OR MORE SYMPTOMS

Direct links to fixes

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R71A PSY

R71H PSY

R71L PSY

R71S PSY

R71W PSY

R81A PSY

R81L PSY

R81W PSY

Document Information

Share your feedback

Need support?