IC99117: FREQUENT ANR0538I MESSAGES WITH HIGH AMOUNT OF "DB2 MFTCH EXEC" TIME DURING EXPIRATION

IBM Tivoli Storage Manager V6.3 Fix Pack 5 (6.3.5.000) Server Downloads

APAR status

• Closed as program error.

Error description

• Frequent messages "ANR0538I A resource waiter has been aborted."
  may be reported in the actlog during client backups.
  This is especially the case during database intensive processes
  like EXPIRATION which do hold required locks for a longer time
   finally causing client backups to  terminate after the
  resourcetimeout has been  reached.
  
  It is not a best practice to let a client backup run
  concurrently with the expiration process but that may happen (as
  not possible otherwise) for a heavily used Tivoli Storage
  Manager server.
  
  A server instrumentation trace shows very slow "DB2 MFtch Exec"
  processing  for the expiration thread.
  
  Analysis has shown the following select as being responsible for
  this high value :
    "select BFID, OFFSET, LENGTH, OWNER,
  DIGEST_TYPE,DIGEST_VALUE,FLAGS,UNCOMP_LENGTH from
  BF_AGGREGATED_BITFILES where SRVID=? and SUPERBFID=? and
  OWNER=?"
  
  A special runstat needs to be done to allow DB2 to use a better
  access plan for this specific table:  BF_AGGREGATED_BITFILES
  
  Tivoli Storage Manager Versions Affected: All platforms  6.1  ,
  6.2 , 6.3 , 7.1
  
  Customer/L2 Diagnostics :
  
  - Servermon script instrumentation trace show the following
  statistics for the expiration thread :
  
  Thread 691414 ExpirationProcessThread parent=691411
  00:16:44.567-->00:41:08.037
   Operation       Count  Tottime  Avgtime  Mintime  Maxtime
  InstTput  Total KB
   ---------------------------------------------------------------
   DB2 Fetch Exec      5    0.082    0.017    0.000    0.042
   DB2 MFtch Exec      1 1391.037 1391.037 1391.037 1391.037
   DB2 Inser Exec 121537   70.183    0.001    0.000    2.580
   DB2 Updat Exec      5    0.082    0.017    0.000    0.042
   DB2 Fetch           5    0.000    0.000    0.000    0.000
   DB2 Commit          1    0.161    0.162    0.161    0.161
   DB2 Reg Prep        2    0.006    0.003    0.000    0.005
   DB2 Reg Exec        3    1.268    0.423    0.000    0.714
   DB2 Reg Fetch       3    0.000    0.000    0.000    0.000
   Tm Lock Wait        1    0.003    0.003    0.003    0.003
   Unknown                  0.644
   ---------------------------------------------------------------
   Total                 1463.470
  
  Please note the considerably high value of time being spent in
  "DB2 MFtch Exec".
  
  Initial Impact: Medium
  
  Additional Keywords:
  BFBF BFBE TSM
  

Local fix

• To reduce the frequency of the ANR0538I messages to appear (may
  be not eliminate all of them as they can still appear due to
  other reasons) caused by the current runstats command, please
  do:
  
  - First be sure that the table BF_AGGREGATED_BITFILES and it's
    indexes does not need reorg.
    This technote explains how to check current reorg status for
    the tables and indexes using "db2 reorgchk" command.
    And it also explains how to do offline reorg if needed.
     "Database size, database reorganization, and performance
     considerations for Tivoli Storage Manager V6 and V7 servers"
     http://www.ibm.com/support/docview.wss?uid=swg21452146
  
  - Run the following runstats command (the command may run when
    server is up and running) :
   db2 "runstats on table TSMDB1.BF_AGGREGATED_BITFILES  ON ALL
   COLUMNS AND COLUMNS ((OWNER,SUPERBFID)) with distribution and
   sampled detailed  indexes all"
   The server should not be halted until the runstats completes.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users.                     *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available.                           *
  * This problem is currently projected to be fixed in levels    *
  * 6.3.6 and 7.1.1.                                             *
  * Note that this is subject to change at the discretion of     *
  * IBM.                                                         *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP