IC89747: CANCELLATION OF CLIENT-SIDE DEDUPLICATION SESSIONS CAN CAUSE SER VER HANG.

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager server can hang after multiple
  client-side deduplication enabled sessions are canceled at or
  near the same time. This can occur with sessions canceled via
  the CANCEL SESSION administrator command, or via a client
  session timeout (i.e. IDLETIMEOUT).
  
  Customer/L2 Diagnostics:
  Dumping the active threads for the dsmserv process will show
  multiple client side deduplication sessions being canceled.
  One thread should include the pkAcquireMutexTracked()
  function, and the other should contain the AdmLockNode()
  function (example below is from the AIX procstack utility):
  
    Thread 1:
    pth_spinlock._global_lock_common()
    pth_mutex._mutex_lock()
    pkmon.pkAcquireMutexTracked()  <-----------------
    smcancel.CancelDedupSession()  <-----------------
    smcancel.CancelSessionNumDedup()
    smcancel.CancelSessionNum()
    smcancel.smCancelSession()
    admcmd.AdmCommandLocal()
    admcmd.admCommand()
    smadmin.SmAdminCommandThread()
    pkthread.StartThread()
  
    Thread 2:
    pth_cond._cond_wait_global()
    pth_cond._cond_wait()
    pth_cond.pthread_cond_wait()
    pkmon.pkWaitConditionTracked()
    tmlock.tmLockTracked()
    admutil.AdmLockNode()  <-----------------
    admnode.admSetNodeSessCanceled()
    smcancel.CancelDedupSession()  <-----------------
    smcancel.CancelSessionNumDedup()
    smcancel.CancelSessionNum()
    smcancel.smCancelSession()
    admcmd.AdmCommandLocal()
    admcmd.admCommand()
    smadmin.SmAdminCommandThread()
    pkthread.StartThread()
  
  Initial Impact:
  Medium
  
  Tivoli Storage Manager Versions Affected:
  All 6.2.4 & 6.3.1 or higher Tivoli Storage Manager servers.
  
  Additional Keywords:
  ZZTSM62 TSM HANG HUNG 76952 DEDUP CLIENT CANCEL SESS
  DEADLOCK MUTEX SPINLOCK LOCKNODE NODE
  

Local fix

• Recycle the dsmserv process to clear the hang.  Avoid
  canceling multiple client deduplication sessions at the
  same time until a fixing level can be applied.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users of   *
  *                 CANCEL SESSION command.                      *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing levels when available.          *
  *                 This problem is currently projected          *
  *                 to be fixed in level 6.2.6 and 6.3.4.        *
  *                 Note that this is subject                    *
  *                 to change at the discretion of IBM.          *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R61Z PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP

• R62Z PSY

     UP

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP

• R63Z PSY

     UP