IT36081: EXCESSIVE AMOUNT OF UNACCESSIBLE SPACE SEEN IN A DIRECTORY CONTAINER POOL

IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads

APAR status

• Closed as program error.

Error description

• On an IBM Spectrum Protect server which uses directory container
  pools, an excessive amount of unaccessible space can be seen
  when using the command 'show sdp'. This is space held by the
  internal defragmentation engine. One consequence of this, is
  that the space held unaccesible is not taken into consideration
  in the overall free space of the storage pool, so a 'Q STG'
  output is not updated to reflect the space held and can give a
  false impression of the amount of free space available. This can
   lead to seeing out of space issues when storing data to a
  directory container pool, although using the  'Q STG' command
  clearly shows ample space available. The unaccessible space
  reported will be a very large amount and could encroach on the
  amount of free space seen within allocated containers
  (DedupCntrUsedSpace) or amount of overall free pool space
  reported (PoolFree) . The amount of unaccessible space held will
   not reduce.
  
  Example :
  
  The command 'Q STG' may show the pool being 42% utilized, but
  the 'show sdp' output displays :
  
  SD Pool YOURCONTAINERPOOL (6):
     Needs Refresh:                        False
     Maximum Size:                         0
     FsCapacity:                    24117149 MB
     FsFreeSpace:                    4114332 MB
     DedupCntrAllocSpace:           19968000 MB
     DedupCntrUsedSpace:             9965720 MB
     DedupCntrFreeSpace:            10002280 MB
     DedupUnaccessibleSpace:         9890563 MB
     NonDedupCntrAllocSpace:            1024 MB
     NonDedupCntrUsedSpace:                3 MB
     NonDedupCntrFreeSpace:             1020 MB
     NonDedupUnaccessibleSpace:            0 MB
     PoolCapacity:                  24083356 MB
     PoolFree:                      14117632 MB
     Pool Unaccessible:              9890563 MB
  
  In this example, the unaccessible space is using 9.4TB whereas
  the amount of space available within allocated dedup containers
  (DedupCntrUsedSpace) is 9.5TB. Overall at a pool level, there is
  actually only 4TB available to use (PoolFree - Pool
  Unaccessible) so in reality the pool has only 4TB free and is
  83% utilized. The way the 'Q STG' output calculates and displays
  the percent utilised is being reviewed in parallel.
  
  
  Customer/Support Diagnostics (If Applicable)
  
  To verify that this APAR is being seen, issue the 'show threads'
  command and review if the output shows thread
  "SdPoolDefragThread" running. Example  :
  
  Thread 59, Parent 1: SdPoolDefragThread, Storage 0, AllocCnt
  6717520 HighWaterAmt 2233040
   tid=140736798566144, ptid=140737352439744, det=1, zomb=0,
  join=0, result=0, sess=0, procToken=0, sessToken=0
   lwp=102460
   Stack traces are disabled by option.
    Awaiting cond descP->defragP->cond (0x0x7fffeee256e0), using
  mutex descP->defragP->mutex (0x0x2679600), at sdcntr.c(3627)
  
  There should be one "SdPoolDefragThread" thread for every
  directory container pool configured. If there are less
  "SdPoolDefragThread" threads running than the amount of
  directory pools configured, then this APAR is being seen as the
  thread has died.
  
  This APAR will report additional messages within the dsmffdc.log
  that this issue is being seen, plus will automatically restart
  the "SdPoolDefragThread" thread which will address the
  unaccessible space seen.
  
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect server versions from 7.1.3 and 8.1 on all
  platforms
  
  Initial Impact:
  Medium
  
  
  Additional Keywords:
  TSM, TS004975661, "storage pool", extents,defrag, engine,
  ANR0522W, ANR0200E
  

Local fix

• 1) Recycle the Spectrum Protect server as soon as possible. This
   will restart the "SdPoolDefragThread" thread again which will
  address the space held, but this may take time for the space to
  reduce.
  
  2) If a recycle cannot be done or there is a need to free the
  unaccessible space immediately, then follow this procedure :
  
  As the instance id owner, or ran within an elevated DB2 CLI on
  Windows, issue :
  
  db2 connect to tsmdb1
  db2 set schema tsmdb1
  
  db2 "delete from sd_fragmented_containers"
  
  Next, run the following command from a Specetrum Protect admin
  CLI :
  
  reset containersize YOURCONTAINERPOOL
  
  This will free the space immediately, but a recycle will still
  need to be performed in order to restart the
  "SdPoolDefragThread" thread. This procedure may need to be done
  once a week if a recycle cannot be performed in that time.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be                                              *
  * fixed in level 8.1.13.  Note that this is subject to change  *
  * at the discretion of IBM.                                    *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:   AIX, Linux, Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81L PSY

     UP

• R81W PSY

     UP

• R81A PSY

     UP