IC78605: Server hang during long-running processes when index reorganization is running

IBM Tivoli Storage Manager (TSM) server V6.2.3.x interim fix downloads
IBM Tivoli Storage Manager (TSM) server V6.1.5.x interim fix downloads

APAR status

• Closed as program error.

Error description

• Problem Description:
  
  Tivoli Storage Manager has certain processes (for example,
  storage pool migration) which run for extended periods of
  time--many hours or even days.  A problem has been found
  in which a long-running server process and index reorganization
  
  can deadlock.  Note, that it is recommended that reorganization
  
  runs when the server is relatively idle.
  
  In general, this APAR can be suspected if index reorganization
  is enabled on the server, if server processes don't finish
  as expected, and if certain conditions don't change over
  30 minutes.
  
  Following are the conditions to check:
  1) Server 'Q PROCESS' output doesn't change appreciably over
     30 minutes.
  2) From a DB2 Command Line Processor window:
     a) db2 connect to tsmdb1
     b) db2 "get snapshot for all applications >application.out"
     c) Examine the application.out file and find the
        "Most recent operation" entry like this:
         Most recent operation = Reorganize
     d) Scroll backwards until finding the "Application status"
        entry. If the status is "Lock-wait", it is waiting for
        the Z lock.  The stanza will look like something like
        this:
  
              Application Snapshot
  Application handle                         = HHHHHH
  Application status                         = Lock-wait
  <Many irrelevant lines>
  Most recent operation                      = Reorganize
  
    e) After 30 minutes, repeat steps b-d, and if the Reorganize
       operation is still in Lock-wait, it is probably
       deadlocked.
  
  3) From the DB2 Command Line Processor window:
    a) db2pd -d tsmdb1 -reorg index >reorg.out
    b) Examine reorg.out, and if there's an in-flight index
       reorg, but the "Cur Count: NNNN" amount doesn't change
       over 30 minutes, it's probably deadlocked.
  
       It will look something like this:
  TbspaceID: 2        TableID: 165
  Schema: TSMDB1   TableName: BF_QUEUED_CHUNKS
  <Irrelevant lines>
  Status: Running
  <Irrelevant lines>
  Cur Count: NNNN                   Max Count: MMMMM
  
  Tivoli Storage Manager Versions Affected:
  Tivoli Storage Manager Server versions 6.1 and 6.2
  
  Initial Impact:
  medium
  
  Additional Keywords:
  zz61 zz62 TSM index reorganization lock contention deadlock
  DB2_KEEPTABLELOCK IC77773
  

Local fix

• 1) Run index reorganization during periods of server inactivity
  
     to avoid this problem altogether.
  2) If you encounter this problem, from the application.out
     file, obtain the application handle HHHHHH for the
     reorganize application.  Be sure that you are using the
     correct application handle.
  
  From a DBP Command Line Processor window:
    a) db2 connect to tsmdb1
    b) Issue the following command in the DB2 Command Line
       Processor Window substituting in the actual application
       handle in for HHHHHH:
          db2 "force application (HHHHHH)"
    c) Because the nature of the command being canceled and that
       the DB2 FORCE APPLICATION command is asynchronous, it
       might take up to 30 minutes for the process to be
       canceled.
    d) After the index reorganization process is canceled, the
       server processes should proceed.
       If not, halt and restart the server
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager server users.     *
  ****************************************************************
  * PROBLEM DESCRIPTION: See error description.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in levels 6.1.5.20, 6.2.3.10,                *
  *                 and 6.2.4.00.                                *
  *                 Note that this is subject to change at the   *
  *                 discretion of IBM.                           *
  ****************************************************************
  *
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, HP-UX, Solaris, Linux and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP