IT16794: CONVERT STGPOOL CHILD PROCESS MAY HANG WHEN TARGET CONTAINER POOL RUNS OUT OF SPACE.

7.1.8.000-TIV-TSMSRV-AIX
7.1.8.000-TIV-TSMSRV-HP-UX
7.1.8.000-TIV-TSMSRV-Linuxppc64
7.1.8.000-TIV-TSMSRV-Linuxs390x
7.1.8.000-TIV-TSMSRV-Linuxx86_64
7.1.8.000-TIV-TSMSRV-SolarisSPARC
7.1.8.000-TIV-TSMSRV-WIN
7.1.8.000-TIV-TSMOC-AIX
7.1.8.000-TIV-TSMOC-Linuxx64
7.1.8.000-TIV-TSMOC-Windows
7.1.8.000-TIV-TSMOC-LinuxPPC64
7.1.8.000-TIV-TSMOC-LinuxPPC64le
7.1.8.000-TIV-TSMOC-LinuxS390

APAR status

• Closed as fixed if next.

Error description

• If a convert storage pool process runs into an out of space
  condition, one or more of the child convert processes may hang.
  Other convert child processes may continue to process although
  they will likely eventually fail (terminating and not hanging)
  with out of space messages. Query process output will show the
  parent process still making progress as long as there is at
  least one child process that is still able to convert/move data.
  If the query process output shows one of the convert child
  processes not making any progress over a given amount of time
  then it may be hung. At some point the query process output will
  only show the parent and this one hung child and neither will
  show any further signs of activity when monitored over time.
  
  Show threads output can be used to validate if a child process
  is in this hung condition. The stack trace (from the show
  threads on AIX or Windows, pstack on Linux) for the
  SdCntrStreamThread will show that SdWaitCQEmpty has gone into
  pkWaitCondition for the session number (procToken and thread
  context process_number) that is hung.
  Example from a Windows server exhibiting this issue where
  process 6 was a child convert process that was no longer making
  progress:
  
  Thread 230, ID 8816 (0x2270): SdCntrStreamThread, procToken=6,
  sessToken=46
     Parent=201, result=0, joining=0, detached=1, zombie=0,
  session=0
  
     Stack trace:
     000000007743C0EA ZwWaitForMultipleObjects()+a
     000007FEFD151430 GetCurrentProcess()+40
     00000000771E06B0 WaitForMultipleObjects()+b0
     000007FED4B2BCD4 pkWaitCondition()+94
     000007FED52E966F SdWaitCQEmpty()+ff
     000007FED52D7C30 WaitForWriteCompletion()+c0
     000007FED52D6A98 SdCntrStreamThread()+408
     000007FED4B27911 startThread()+141
     000007FEEBF04F7F beginthreadex()+107
     000007FEEBF05126 endthreadex()+192
     00000000771E59BD BaseThreadInitThunk()+d
     000000007741A2E1 RtlUserThreadStart()+21
  
     Thread context:
     COMMMETHOD: Tcp/Ip
     COMMAND: CONVERT STGPOOL
     THREAD_TYPE: PROCESS
     PROCESS_DESC: CONVERT STGPOOL
     PROCESS_NUMBER: 6
     SESSION_TYPE: ADMIN
  
   3  Convert Stgpool    Converting storage pool FILEDEDUP to
  directory-container storage pool CONTAINERPOOL. Volumes
  Converted: 305 of 798, Volumes Failed: 4, Converted Files:
  179223, Converted Bytes: 3.89 TB, Skipped Files: 12, Skipped
  Bytes: 1.30 GB, Total Bytes Transferred: 145.56 GB
   6  Convert Stgpool    Converting storage pool FILEDEDUP to
  directory-container storage pool CONTAINERPOOL. Converted Files:
  0, Converted Bytes: 0 B of 2.07  TB, Skipped Files: 0, Skipped
  Bytes: 0 B, Total Bytes Transferred: 0 B, Current input volume:
  H:\TSMSERVER1\VOL123, Elapsed time: 0 Days, 17 Hours, 52
  Minutes.
  
  NOTE that the process 6 above shows it has been processing for
  17+ hours with no work done.
  
  Tivoli Storage Manager Versions Affected: IBM Spectrum Protect
  7.1.6 and above on all platforms
  
  Initial Impact: Medium
  
  Additional Keywords:  TSM STG
  

Local fix

• Add more space to the target container storage pool so that the
  out of space issues are not encountered and this hang will be
  avoided.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users of container storage   *
  * pools.                                                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * This problem is projected to be fixed in a future version of *
  * the Tivoli Storage Manager server.  Note that this is        *
  * subject to change at the discretion of IBM.                  *
  *                                                              *
  * Affected platforms:  AIX, Solaris, Linux, and Windows.       *
  ****************************************************************
  

Problem conclusion

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

Applicable component levels