IC75496: SERVER HANGS DUE TO DB2 AND DSMSERV DEADLOCK SITUATION OVER VOLUME DELETION ACTIVITY

IBM Tivoli Storage Manager (TSM) server V6.1.5.x interim fix downloads

APAR status

• Closed as program error.

Error description

• Server can appear to hang under specific circumstances when
  volume deletions are taking place. The exact scenario for the
  hang to occur is as follows:
  
  A reclamation process encounters a volume that has no data on it
  so a direct deletion is attempted. This deletion request is
  unique to the reclamation process and entails a sequence of
  steps that do not follow the order required by the global
  deletion routines. At the same time, the standard deletion
  routine is actively attempting to delete some scratch volumes
  and is following the correct sequence of steps which results in
  contention with the reclamation process. The callstacks for each
  thread will look similar to the following:
  
  RECLAMATION PROCESS THREAD
  #0  0x00000039e9ad5337 in semop () from /lib64/libc.so.6
  #1  0x00002ac9885581b7 in sqloSSemP () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #2  0x00002ac9884c1f1a in sqlccipcrecv(SQLCC_COMHANDLE_T*,
  SQLCC_COND_T*) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #3  0x00002ac9884c6523 in sqlccrecv () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #4  0x00002ac988516d71 in sqljcReceive(sqljCmnMgr*) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #5  0x00002ac98852ed07 in sqljrReceive(sqljrDrdaArCb*,
  db2UCinterface*, bool) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #6  0x00002ac988c48c46 in sqljrDrdaArExecute(db2UCinterface*,
  UCstpInfo*) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #7  0x00002ac988610989 in CLI_sqlExecute(CLI_STATEMENTINFO*,
  CLI_ERRORHEADERINFO*) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #8  0x00002ac9886a80ab in SQLExecute2(CLI_STATEMENTINFO*,
  CLI_ERRORHEADERINFO*) () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #9  0x00002ac9886bd05b in SQLExecute () from
  /home/itsm/sqllib/lib64/libdb2.so.1
  #10 0x000000000077b42a in RdbPrepareAndExecuteStmt ()
  #11 0x000000000077eaac in tbCliSRInsDel ()
  #12 0x0000000000bf0c38 in SsRemoveVolume ()
  #13 0x0000000000bf0f6a in ssDeleteVolume ()
  #14 0x00000000006d3849 in EndMoveVolume ()
  #15 0x00000000006d422e in AfRclmOnsiteVols ()
  #16 0x00000000006d7152 in AfRclmVolumeThread ()
  #17 0x0000000000ccbf3b in StartThread ()
  
  GLOBAL DELETION THREAD
  #1  0x0000000000ccd67b in pkWaitConditionTracked ()
  #2  0x0000000000c23860 in tmLockTracked ()
  #3  0x000000000074ff07 in LockVolumeTypeId ()
  #4  0x000000000074ffcd in LockVolumeType ()
  #5  0x000000000075035c in IcLockVolumeHistory ()
  #6  0x000000000074bd6f in IcvhAddVolume ()
  #7  0x000000000074d1fb in icAddDeletedStgVol ()
  #8  0x0000000000b75c4d in DeleteScratch ()
  #9  0x0000000000b78884 in AsScratchDeleteThread ()
  #10 0x0000000000ccbf3b in StartThread ()
  #11 0x00000039ea60673d in start_thread () from
  /lib64/libpthread.so.0
  
  They key functions for this hang condition are DeleteScratch and
  SsRemoveVolume.
  
  TSM Versions Affected:
  Tivoli Storage Manager server 6.x on all platforms.
  
  Initial Impact:
  Medium
  
  Additional Keywords:
  zz61 zz62 deadlock hang wait reclaim contention
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available.           *
  *                 This problem is currently projected to be    *
  *                 fixed in levels 6.1.5.1, and 6.2.3     *
  *                 Note that this is subject to change at       *
  *                 the discretion of IBM.                       *
  ****************************************************************
  *
  

Problem conclusion

• The described problem has been resolved.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP