IC71921: NDMP BACKUP NODE WITH TOC=YES CAN CREATE ORPHANED TRANSACTIONS T HAT PIN THE ACTIVE LOG

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager server's active log can become
  pinned by orphaned and uncommitted transactions.  Externally,
  this will cause continuous growth of the Server's Active Log
  over a period of days or weeks.  Eventually, this can cause
  the Active Log to fill and the Server to stop.
  .
  The orphaned transactions are created when an NDMP BACKUP NODE
  operations fails.  One error condition that this is known to
  occur is when an NDMP BACKUP NODE operation fails because
  there is already an existing NDMP operation in progress for
  the specified file space.  For example:
    ANR1095E Data mover operation for node CEL1 and file space
             /fs10 cannot be started an operation involving
             this node and file space is already in progress.
  .
  Customer/L2 Diagnostics:
  Collect and review SHOW TXNT and SHOW THREADS output from the
  Server.  Search the transactions output (SHOW TXNT) for
  transactions involving NDMP temporary tables.  The associated
  threads for these transactions normally need to exist in the
  thread output (SHOW THREADS).  If they do not, this APAR
  applies.
  .
  For example:
  .
  SHOW TXNT output:
  Tsn=0:619110943, Resurrected=False, InFlight=True,
     Distributed=False, Persistent=True, Addr 1ee55e38
    ThreadId=245351, Timestamp=10/10/10 10:10:10,
     Creator=tbtbl.c(6336) Participants=1, summaryVote=Commit
    EndInFlight False, endThreadId 245351, tmidx 0 0,
     processBatchCount 0, mustAbort False.
      Participant DB: voteReceived=False, ackReceived=False
        DB: Txn 111fd7c78, ReadOnly(NO), connP=1120adf18,
            applHandle=29592, openTbls=3:
        DB: --> OpenP=115051f98 for
                tempTable=(NDMP.Nodes(TEMP),TEMP_NODE).
        DB: --> OpenP=1164c8b38 for table=Filespaces.
        DB: --> OpenP=117809ad8 for table=Nodes.
  .
  As per the output, the associated thread for this transaction
  is ThreadId=245351. The SHOW THREADS output should list
  thread 245351.  If it does not, this APAR applies.
  .
  An additional verification that this error has occurred is to
  use the SHOW TOCSETS command to look for orphaned TOC sets in
  the server:
  .
  SHOW TOCSETS
  ANR2017I Administrator SERVER_CONSOLE issued command:
    show tocsets
  Active toc sets:
  ---------------------------------------
    magic=0x53434f54 (TOCS)
    seqNum=3, type=build state=TocSessBuilding
    node=CEL1(17), fsId=6, dataType=0
    mergeActive=False
    state=1, currentToc=0
    totalEntries=0, procEntries=0
  lastUsed=255-255-2155 255:255:255
  .
  If there are no active BACKUP NODE operations for the node
  and filespace Identifier and the TOC set state is
  "TocSessBuilding" then this is an orphaned TOC set with
  orphaned transactions.
  .
  Initial Impact:
  Medium.
  .
  Tivoli Storage Manager Versions Affected:
  All 6.1 and 6.2 Tivoli Storage Manager Servers.
  .
  Additional Keywords:
  TSM DSMSERV ZZ61 ZZ62 TRANSACTION THREADS ORPHAN ORPHANED
  TXN TXNT ACTIVE LOG GROWTH PIN PINNED PINNING NDMP TOC
  T-O-C NAS BACKUP NODE ANR1095E
  .
  

Local fix

• Recycle the Server instance to clear the pin and temporarily
  resolve the issue.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager Users of NDMP     *
  *                 backup with TOC creation.                    *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION                   *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be         *
  *                 fixed in levels 6.1.5 and 6.2.3 . Note       *
  *                 that this is subject to change at the        *
  *                 discretion of IBM.                           *
  ****************************************************************
  *
  

Problem conclusion

• The problem is fixed.
  Affected Platforms:  AIX,HP-UX, Sun Solaris, Linux, and
  Windows
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP