IC79026: IDENTIFY DUPLICATES MIGHT NOT PROCESS ALL ELIGIBLE VOLUMES IN A TIMELY MANNER

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager Server identify duplicates process
  might experience delays in processing eligible volumes.
  The symptom will be that a process is shown as active
  but the amount of data processed will not increase over time.
  This can happen when there is at least one additional operation
  that has acquired a volume that Identify istrying to process.
  For instance, if an Identify process is trying to acquire volume
  A and that volume is in the process of being read/written to by
  reclamation, the Identify process will actively wait for access
  to that volume. The problem comes into play when there are many
  volumes to be processed within the deduplicated storage pool.
  The Identify process could be delayed for minutes/hours while
  waiting for access to the volume thus delaying potential
  processing of other eligible volumes.
  .
  Customer/L2 Diagnostics:
  There are a couple of things to look for here:
  1) If the process is active but not incrementing the file/byte
      counts over a long period of time, this APAR could be in
      play. Proceed to #2.
  .
  2) Issue SHOW THREADS (or pstack/gstack for platforms
      other than AIX) and SHOW DEDUPTHREAD. The
      callstack for the Identify process (BfDedupVolumeThread)
      will show a wait in pkDelayThread:
        Thread 22 (Thread 0x4153d940 (LWP 31024)):
        #0  0x00000030e489a3f1 in nanosleep () from
  /lib64/libc.so.6
        #1  0x00000030e489a214 in sleep () from /lib64/libc.so.6
        #2  0x0000000000cf2f0f in pkDelayThread ()
        #3  0x000000000069f80e in BfDedupVolumeThread ()
        #4  0x0000000000cf305b in StartThread ()
        #5  0x00000030e540673d in start_thread () from
  /lib64/libpthread.so.0
        #6  0x00000030e48d3f6d in clone () from /lib64/libc.so.6
  .
      The SHOW DEDUPTHREAD output will show the thread as
       busy but no bitfile id being processed.
  .
      ---> Volume Dedup Control Thread 82544:
           Process Number: 54
           Current Volume ID: 3728
           Current bitfile ID: 0
           State: busy, Cancelled: False
  .
  Additional Keywords: bfdedupvolumeagent bfdedupvolumethread
  deduplication dedupe dedup slow hung hang
  performance degradation large file huge lock wait
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available.           *
  *                 This problem is currently projected to be    *
  *                 fixed in level 6.3.1.                        *
  *                 Note that this is subject to change at       *
  *                 the discretion of IBM.                       *
  ****************************************************************
  *
  

Problem conclusion

• The described problem has been resolved.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP

• R63A PSY

     UP

• R63H PSY

     UP

• R63L PSY

     UP

• R63S PSY

     UP

• R63W PSY

     UP