IT37113: BACKUP TO DIRECTORY CONTAINER STGPOOLS IS SLOW, SESSION HANG AND ACTIVE LOG FILL UP SINCE UPGRADE TO 8.1.11.100

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• After an upgrade or new installation of  IBM Spectrum Protect
  Server version 8.1.11.100, some backup/archive sessions looks
  not progressing, and active log space usage increase at the
  server.
  A CANCEL SESSION does not stop the sessions. The only way to
  clean these sessions and free active log is to do a server
  restart.
  
  One signature of that problem is seen in a servermon data
  collection.
  Some threads are waiting on condition named "preliminaryCQCond"
  and never come out of this, one example below in thread
  "AsyncWriteThread":
  
  Thread 49061, Parent 49034: AsyncWriteThread, Storage 26883488,
  AllocCnt 478605 HighWaterAmt 26883536
  tid=1991b, ptid=14f00, det=0, zomb=0, join=0, result=0, sess=0,
  procToken=0, sessToken=0
   Stack trace:
     0x090000000057e180 _cond_wait_global
     0x090000000057ee08 _cond_wait
     0x090000000057f7ac pthread_cond_wait
     0x000000010000b2b4 pkWaitConditionTracked
     0x0000000100a9e464 IPRA.$IsTxnFlushRequired
     0x0000000100a9c0ac SdAsyncWrite
     0x0000000100b0f170 AsyncWriteThread
     0x0000000100a7449c PcConsumerThread
     0x0000000100b0f0b4 AsyncWriteStackEntry
     0x0000000100011530 StartThread
  
   Awaiting cond sessP->preliminaryCQCond (0x1f8f231a0), using
  mutex sessP->preliminaryCQMutex (0x1405491c0), at sdio.c(3629)
  Thread context:
  
    SRC_STRATEGY: CONTAINER
    NODE_NAME: NODENAME
  
  Another symptom seen is when that condition wait happens for
  "SdCQSinkThread", as seen here:
  
   Thread 8921, Parent 8907: SdCQSinkThread, Storage 12539280,
  AllocCnt 220853 HighWaterAmt 26884672
      tid=18ed9, ptid=1fbcb, det=1, zomb=0, join=0, result=0,
  sess=0, procToken=0, sessToken=4088
       Stack trace:
         0x090000000053c360 _cond_wait_global
         0x090000000053cef8 _cond_wait
         0x090000000053dbe0 pthread_cond_wait
         0x000000010000b2b4 pkWaitConditionTracked
         0x0000000100a94204 SdFlushSessControls
         0x0000000100a97278 SdCQSinkThread
         0x0000000100011530 StartThread
       Holding mutex sessP->cqControlMutex (0x127a05560), acquired
  at sdbuf.c(1112)
       Awaiting cond sessP->preliminaryCQCond (0x17c03c9d0), using
  mutex sessP->preliminaryCQMutex (0x129cd6ea0), at sdbuf.c(2375)
      Thread context:
        COMMMETHOD: SSL
        PORT: 11.8.7.6:51320
        SRC_STRATEGY: CONTAINER
        SESSION: 4088
        SRC_STGPOOL_NAME: STGXXX
        THREAD_TYPE: SESSION
        SESSION_TYPE: NODE
        NODE_NAME: NODEXXX
  
  IBM Spectrum Protect Versions Affected:
    8.1.11.100 on all supported platforms
  
  
  | MDVREGR 8.1.11.100-TIV_5698MSV |  IT35326
  
  
  Additional Keywords:  TS005567524  hung freeze log  full
                       num_log_span
  

Local fix

• . Roll back to previous level.
  Or
   . Uprade to 8.1.12.0 level
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP