IC99758: SLOW DATA MOVEMENT FROM A DEDUPLICATION ENABLED STORAGE POOL

APAR status

• Closed as program error.

Error description

• Processes such as migration, reclamation, move data, and move
  nodedata will appear to be hanging due to the wrong table index
  being used during delete operations against the storage pool.
  Query process will show that the process has not processed any
  files for several hours.  The files do not have to be
  deduplicated, but rather just stored in a deduplication enabled
  storage pool to encounter this error.
  A trace of AFMOVE shows delete processing of a single bitfile
  taking a very long time:
  09:23:26.350
  [44][aferase.c][950][AfDeleteBitfileFromPool]:Bitfile is
  not an aggregate
  09:23:26.352 [44][afmove.c][7620][MoveBatch]:Deleting moved
  bitfile
  6614807215 in poolId=7.
  09:23:26.352
  [44][aferase.c][658][AfDeleteBitfileFromPool]:Entering for
  bitfile: 6614807215, pool: 7, uncatalog: T, check: T,
  invalidate: F
  delAggr T
  09:24:27.773
  [44][aferase.c][950][AfDeleteBitfileFromPool]:Bitfile is
  not an aggregate
  09:24:27.775 [44][afmove.c][7620][MoveBatch]:Deleting moved
  bitfile
  6614807399 in poolId=7.
  09:24:27.775
  [44][aferase.c][658][AfDeleteBitfileFromPool]:Entering for
  bitfile: 6614807399, pool: 7, uncatalog: T, check: T,
  invalidate: F
  delAggr T
  09:25:27.971
  [44][aferase.c][950][AfDeleteBitfileFromPool]:Bitfile is
  not an aggregate
  09:25:27.973 [44][afmove.c][7620][MoveBatch]:Deleting moved
  bitfile
  6614807577 in poolId=7.
  09:25:27.973
  [44][aferase.c][658][AfDeleteBitfileFromPool]:Entering for
  bitfile: 6614807577, pool: 7, uncatalog: T, check: T,
  invalidate: F
  delAggr T
  An explain file generated from DB2 against a select statement
  can also confirm that the problem is being encountered based on
  which index is shown to be used.  The following steps can be
  taken to create the explain file:
  db2 connect to tsmdb1
  db2 set schema tsmdb1
  db2 set current explain mode explain
  db2 "SELECT be1.bfid, be1.linkbfid FROM
  TSMDB1.BF_BITFILE_EXTENTS be1
  LEFT JOIN TSMDB1.bf_bitfile_extents be2 ON (be1.srvid=be2.srvid
  AND
  be1.poolid=be2.poolid AND be1.bfid=be2.linkbfid) WHERE
  (be1.SRVID=? AND
  be1.POOLID=? AND be1.SUPERBFID=? AND be2.SRVID IS NULL) ORDER BY
  be1.bfid OPTIMIZE FOR 1000 ROWS WITH CS"
  db2exfmt -d tsmdb1 -1 -o expl.out
  If BFBE_BFBF_FK_INDEX is being used then this error is not being
  encountered.  If, BFBE_NDX is being used, then this APAR is
  being encountered.
  Tivoli Storage Manager Versions Affected:
  7.1.x.x Tivoli Storage Manager Servers
  Initial Impact: Medium
  Additional Keywords:
  TSM
  

Local fix

• Run a reorganization on the BF_BITFILE_EXTENTS table
  followed by a runstats.  This may not always fix the issue.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All Tivoli Storage Manager server users with deduplicated    *
  * storage pools.                                               *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 7.1.1. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• dsmserv
  

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R71A PSY

     UP

• R71H PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP