IT31042: EXPIRATION SLOW WHEN PROCESSING LARGE FRAGMENTED FILES IN CONTAINER STORAGE POOLS

8.1.10.000-IBM-SPOC-WindowsX64
8.1.10.000-IBM-SPOC-Linuxx86_64
8.1.10.000-IBM-SPOC-Linuxs390x
8.1.10.000-IBM-SPOC-LinuxPPC64le
8.1.10.000-IBM-SPOC-AIX
8.1.10.000-IBM-SPCMS-WindowsX64
8.1.10.000-IBM-SPCMS-WindowsI32
8.1.10.000-IBM-SPCMS-Linuxx86_64
8.1.10.000-IBM-SPSRV-WindowsX64
8.1.10.000-IBM-SPSRV-Linuxx86_64
8.1.10.000-IBM-SPSRV-Linuxs390x
8.1.10.000-IBM-SPSRV-Linuxppc64le
8.1.10.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack (V8.1.10) Downloads

APAR status

• Closed as program error.

Error description

• Customer/L2 Diagnostics:
  
  In the activity log, look for the ANR0165I and ANR0166I
  messages. The inventory file expiration for a filespace of a
  node with large fragmented  files can take a long time to
  complete.  Only a few files might have been processed during
  that period
  
  Run the server monitoring perl script servermon.pl attached to
  the following technote:
  Perl script to collect IBM Spectrum Protect server monitoring
  data V3.1
  http://www-01.ibm.com/support/docview.wss?uid=swg21432937
  
  The collectstmt instrumentation output shows for the same thread
  a long running delete  with a small a number of executions and a
  shorter running insert with a high number of executions:
   [...]
   Execution Time: 583.510 sec, Number of Executions: 10,
  RowsReturned: 0
   DELETE  FROM "TSMDB1"."SD_RECON_ORDER"  WHERE (OBJID=?) --270
   [...]
   Execution Time: 150.595 sec, Number of Executions: 141453,
  RowsReturned: 0
   INSERT INTO "TSMDB1"."SD_REFCOUNT_UPDATES"
  (POOLID,CHUNKID,BATCHID,UPDCOUNT,PENDINGID) VALUES(?,?,?,?,?)
  --270
   [...]
  
  The instrumentation trace will show a low count for the "DB2
  Delet Exec" and high count for the "DB2 Inser Exec"
   Thread 270 BfDeleteFragmentThread ...
   Operation       Count  Tottime  Avgtime  Mintime  Maxtime
  InstTput  Total KB
   ---------------------------------------------------------------
  -------------
   DB2 Fetch Exec     10    0.026    0.003    0.000    0.014
   DB2 MFtch Exec     10    0.006    0.001    0.000    0.000
   DB2 Inser Exec 154185  190.003    0.001    0.000    0.103
   DB2 Delet Exec     29  534.159   18.419    0.000   71.298
   DB2 Updat Exec      9    0.018    0.002    0.000    0.014
   [...]
  
  In the "show" file the thread has "sdDestroyAll" in the call
  stack
  Thread 270, Parent 1: BfDeleteFragmentThread, Storage 0,
  AllocCnt 9966962 HighWaterAmt 63840
   tid=d60e, ptid=1, det=0, zomb=0, join=0, result=0, sess=0,
  procToken=0, sessToken=0
    Stack trace:
      0x0900000000121828 recv
      0x0900000007cf4518
  tcprecv__FP17SQLCC_COMHANDLE_TiPcT2UsT5P21SQLCC_TCPCONNHANDLE_TP
  12SQLCC_COND_TUiPi
      0x0900000007cf04a8
  sqlcctcprecv__FP17SQLCC_COMHANDLE_TP12SQLCC_COND_T
      0x0900000007cd8498 sqlccrecv
      0x0900000008257f50 sqljcReceive__FP10sqljCmnMgr
      0x0900000008204e9c
  sqljrDrdaArExecute__FP14db2UCinterfaceP9UCstpInfo
      0x090000000844046c
  CLI_sqlExecute__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
      0x09000000084c8e90
  SQLExecute2__FP17CLI_STATEMENTINFOP19CLI_ERRORHEADERINFO
      0x09000000084c3758 SQLExecute
      0x000000010018eb44 IPRA.$RdbPrepareAndExecuteStmt
      0x0000000100188f14 IPRA.$RdbCliInsert
      0x0000000100188638 tbCliSRInsDel
      0x00000001013cca38 SdQueueRefCountUpd
      0x0000000100428e20 IPRA.$ProcessDedupChunkForDelete
      0x000000010042a5ac sdDestroyAll
      0x0000000100bc2a54 bfDestroy
      0x0000000100bb2248 CleanupDeletedSuperAggregate
      0x0000000100baea40 BfDeleteFragmentThread
      0x00000001006823a4 PcConsumerThread
      0x0000000100bae854 BfDeleteFragmentStackTrackEntry
      0x00000001000104c8 StartThread
    Holding mutex txnP->mutex (0x130f156b0), acquired at
  tbcli.c(1160)
  
  
  Initial Impact: High
  
  
  IBM Spectrum Protect versions affected:
  Server 7.1.3 and higher, 8.1.x and higher on all supported
  platforms
  
  
  Keywords:
  TSM  TS002356897  expiration
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.9. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  *                                                              *
  * Set the new option ContainerDelDeferFirstFrag to TRUE by     *
  * using the SETOPT command. Beginning with the 8.1.10 level,   *
  * the default for this option will change to TRUE.             *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP