IT40923: TRANSACTION ABORT DURING STGRULE COPY TO A TAPEPOOL ORPHANS THREAD AND LEAVES VOLUME IN USE BY COMPLETED PROCESS

8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX
8.1.16.000-IBM-SPCMS-WindowsX64
8.1.16.000-IBM-SPCMS-WindowsI32
8.1.16.000-IBM-SPCMS-Linuxx86_64
8.1.16.000-IBM-SPOC-WindowsX64
8.1.16.000-IBM-SPOC-Linuxx86_64
8.1.16.000-IBM-SPOC-Linuxs390x
8.1.16.000-IBM-SPOC-LinuxPPC64le
8.1.16.000-IBM-SPOC-AIX
8.1.16.000-IBM-SPSRV-WindowsX64
8.1.16.000-IBM-SPSRV-Linuxx86_64
8.1.16.000-IBM-SPSRV-Linuxs390x
8.1.16.000-IBM-SPSRV-Linuxppc64le
8.1.16.000-IBM-SPSRV-AIX

APAR status

• Closed as program error.

Error description

• If a stgrule copy to a tapepool job receives ANR0102E,  the
  worker thread may abort and  leave stuck tape after the process
  finishes. The stuck tape requires restart server to release
  
  Support/Customer diagnostics:
  
  1: A stgrule copy process for container pool failed:
  
  ANR0102E bfsaggr.c(3556): Error 1202 inserting row in table
  "BF.Super.Aggregates". (PROCESS: 99, JOB: 3914)
  ANR2183W sctierbatch.c(2717): Transaction 0:22688557 was
  aborted. (PROCESS: 99, JOB: 3914)
  ANR0515I Process 99 closed volume volume1. (PROCESS: 99, JOB:
  3914)
  ANR0220I Copying process 99 for storage pool CONTPOOL has
  completed. (PROCESS: 99, JOB: 3914)
  ANR0986I Process 99 for Copy Storage Pool (Worker)running in the
  BACKGROUND processed 1,150 items for atotal of 257,944,350,300
  bytes with a completion state of FAILURE at 14:28:10. (PROCESS:
  99, JOB: 3914)
  2: the following will be logged in dsmffdc.log for the problem
  object:
  
  [04-26-2022 12:02:13.592][ FFDC_GENERAL_SERVER_ERROR ]:
  (bfutil.c:21412) rc=1031 is fatal for SD streaming
  [04-26-2022 12:02:13.705][ FFDC_GENERAL_SERVER_ERROR ]:
  (bfutil.c:21412) rc=1031 is fatal for SD streaming
  [04-26-2022 12:02:13.708][ FFDC_GENERAL_SERVER_ERROR ]:
  (sctierbatch.c:3544) Tier operation skipped objId 2063780564: rc
  1031
  3:
  
  after the process end.  Tape volume1 remain in use and the
  thread associate to process 99 remains active
  -Query process :
  
  ANR0944E QUERY PROCESS: No active processes found.
  
  -q mount f=d
  ANR8330I 3592 volume 700166 is mounted R/W in drive FR3-DR02
  (/dev/rmt15), status: IN USE ((PROCESS: 99 ..
  -show thread
  
  
  Thread 667, Parent 649: SdCntrStreamThread, Storage 0, AllocCnt
  105 HighWaterAmt 2451456
  tid=14ef9, ptid=18be7, det=1, zomb=0, join=0, result=0, sess=0,
  procToken=9, sessToken=0
   Stack trace:
     0x0900000000571220 _cond_wait_global
  
     0x0900000000571ea8 _cond_wait
  
     0x090000000057284c pthread_cond_wait
  
     0x000000010000b2b4 pkWaitConditionTracked
  
     0x0000000100657734 PvrPrepareClose
  
     0x00000001006528b4 pvrClose
  
     0x0000000100317c48 AsCloseVol
  
     0x00000001008279cc IPRA.$ReleaseSessionEnding
  
     0x0000000100826bd0 AsEndSessionTracked
  
     0x00000001008ee038 ssEndSession
  
     0x0000000100320ac0 bfEndSession
  
     0x00000001003585ac sdEndSession
  
     0x00000001007b7ad4 IPRA.$CleanupCntrStreamThread
  
     0x00000001007ac79c SdCntrStreamThread
  
     0x0000000100011470 StartThread
  
   Awaiting cond agentP->readyCond (0x116cce460), using mutex
  agentP->mutex (0x2052eb408), at pvr.c(13710)
  Thread context:
  
    COMMAND: START STGRULE
    PROCESS_NUMBER: 99
    PROCESS_DESC: Copy Storage Pool (Worker)
    THREAD_TYPE: PROCESS
    JOB_ID: 3914
  
  A restart server is required to clear above orphaned thread and
  stuck tape
  
  Platform Version affected
  Spectrum protect server v8.1.11 and above on all supported
  platform
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.15.100 and  8.1.16. Note *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels