Direct links to fixes
8.1.1.000-IBM-SPSRV-Linuxx86_64
8.1.1.000-IBM-SPSRV-AIX
8.1.1.000-IBM-SPSRV-Linuxs390x
8.1.1.000-IBM-SPSRV-WindowsX64
8.1.1.100-IBM-SPOC-AIX
8.1.1.100-IBM-SPOC-Linuxs390x
8.1.1.100-IBM-SPOC-Linuxx86_64
8.1.1.100-IBM-SPOC-WindowsX64
7.1.8.000-TIV-TSMSRV-AIX
7.1.8.000-TIV-TSMSRV-HP-UX
7.1.8.000-TIV-TSMSRV-Linuxppc64
7.1.8.000-TIV-TSMSRV-Linuxs390x
7.1.8.000-TIV-TSMSRV-Linuxx86_64
7.1.8.000-TIV-TSMSRV-SolarisSPARC
7.1.8.000-TIV-TSMSRV-WIN
7.1.8.000-TIV-TSMOC-AIX
7.1.8.000-TIV-TSMOC-Linuxx64
7.1.8.000-TIV-TSMOC-Windows
7.1.8.000-TIV-TSMOC-LinuxPPC64
7.1.8.000-TIV-TSMOC-LinuxPPC64le
7.1.8.000-TIV-TSMOC-LinuxS390
IBM Spectrum Protect Server V8.1 Fix Pack 1 (V8.1.1) Downloads
APAR status
Closed as program error.
Error description
Executing an UPDATE STGPOOLDIR command with ACCESS=DESTROYED or an AUDIT CONTAINER command with ACTION=MARKDAMAGED when run at the stgpool or stgpooldir level can be a very long running command. Performance enhancements are needed to improve the overall processing time of this command. While either of these two commands are running a status monitoring thread will end up with a lock waiting on the command to complete. These threads will lead to the hanging of other commands such a query stg pool that process container related information. While in this state query/show output will reveal the following (on a Linux system where the stack trace has to be obtained from pstack output and indexed using the lwp from the show thread output): Show locks: LockDesc: Type=57001(mon status), NameSpace=0, SummMode=sLock, Key='' Holder: (monstats.c:2929 Thread 238) Tsn=0:10167353, Mode=sLock Waiter: (monstats.c:9294 Thread 38) Tsn=0:10174315, Mode=xLock slot -> 2116: An ANR0538I A resource waiter has been aborted message will be seen at the resourcetimeout for the waiting thread. Query session: Sess Comm. Sess Wait Bytes Bytes Sess Platform Client Name Number Method State Time Sent Recvd Type ------- ------ ------ ------ ------- ------- ------- -------- ----------------- 107,362 Tcp/Ip Run 0 S 554.1 M 826.7 K Admin Windows IBM-OC-DELHI 126,153 Tcp/Ip Run 0 S 119.1 K 788 Admin Linux TODDO x86-64 126,192 Tcp/Ip Run 0 S 11.8 K 406 Admin Linux TODDO x86-64 tid=140655277889280, ptid=140654646974208, det=1, zomb=0, join=0, result=0, sess=126153, procToken=0, sessToken=126153 lwp=14906 Awaiting cond descP->outReady (0x0x2b2e4f0), using mutex OUTV->mutex (0x0x22e60f8), at output.c(3720) Thread context: COMMMETHOD: Tcp/Ip THREAD_TYPE: SESSION SESSION_TYPE: ADMIN ADMIN_NAME: TODDO Thread 1065290, Parent 1065053: SmAdminCommandThread, Storage 469002, AllocCnt 1957 HighWaterAmt 473454 tid=140646789461760, ptid=140655277889280, det=0, zomb=0, join=0, result=0, sess=0, procToken=0, sessToken=126153 lwp=19611 Holding mutex descP->mutex (0x0x2731de8), acquired at sddir.c(1174) Thread context: COMMAND: UPDATE STGPOOLDIRECTORY COMMMETHOD: Tcp/Ip THREAD_TYPE: SESSION SESSION_TYPE: ADMIN ADMIN_NAME: TODDO Show thread output for 126192 (parent and child): Thread 1065550, Parent 224: psSessionThread, Storage 91806, AllocCnt 295 HighWaterAmt 196981 tid=140646807357184, ptid=140654646974208, det=1, zomb=0, join=0, result=0, sess=126192, procToken=0, sessToken=126192 lwp=24944 Awaiting cond descP->outReady (0x0x2b3a420), using mutex OUTV->mutex (0x0x22e60f8), at output.c(3720) Thread context: COMMMETHOD: Tcp/Ip THREAD_TYPE: SESSION SESSION_TYPE: ADMIN ADMIN_NAME: TODDO Thread 1065604, Parent 1065550: SmAdminCommandThread, Storage 128582, AllocCnt 91 HighWaterAmt 166196 tid=140646287406848, ptid=140646807357184, det=0, zomb=0, join=0, result=0, sess=0, procToken=0, sessToken=126192 lwp=25977 Holding mutex descP->tableMutex (0x0x7feba40090b8), acquired at output.c(1920) Awaiting cond descP->space.condition (0x0x7fed0bceeed0), using mutex descP->space.mutex (0x0x272ca78), at sdpool.c(4847) Thread context: COMMAND: QUERY STGPOOL COMMMETHOD: Tcp/Ip THREAD_TYPE: SESSION SESSION_TYPE: ADMIN ADMIN_NAME: TODDO Pstack output for session 126153/lwp 19611: Thread 6 (Thread 0x7feae1ecb700 (LWP 19611)): #0 0x00000037c94eb197 in semop () from /lib64/libc.so.6 #1 0x00007fed0eaff0ac in sqloSSemP () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #2 0x00007fed0e9c4e10 in sqlccipcrecv(SQLCC_COMHANDLE_T*, SQLCC_COND_T*) () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #3 0x00007fed0e9c7e40 in sqlccrecv () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #4 0x00007fed0ea6c289 in sqljcReceive(sqljCmnMgr*) () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #5 0x00007fed0f38e0a4 in sqljrDrdaArExecute(db2UCinterface*, UCstpInfo*) () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #6 0x00007fed0e87c7b8 in CLI_sqlExecute(CLI_STATEMENTINFO*, CLI_ERRORHEADERINFO*) () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #7 0x00007fed0e8ce11c in SQLExecute2(CLI_STATEMENTINFO*, CLI_ERRORHEADERINFO*) () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #8 0x00007fed0e8ca64f in SQLExecute () from /home/tsminst1/sqllib/lib64/libdb2.so.1 #9 0x0000000000c72172 in tbRegExecEx () #10 0x0000000000cd7c18 in SdUpdateDestroyedStateInAuditTable () #11 0x0000000000d12857 in SdUpdateStgpooldirAccess () #12 0x0000000000cd91c1 in sdUpdateStgPoolDir () #13 0x0000000000698034 in AdmUseExtCmdTab () #14 0x0000000000554a98 in AdmCommandLocal () #15 0x00000000005561c9 in admCommand () #16 0x0000000000d59652 in SmAdminCommandThread () #17 0x00000000010e7452 in StartThread () #18 0x00000037c98079d1 in start_thread () from /lib64/libpthread.so.0 #19 0x00000037c94e8b6d in clone () from /lib64/libc.so.6 Pstack output for sessoin 126192/lwp 25977: Thread 3 (Thread 0x7feac3fff700 (LWP 25977)): #0 0x00000037c980b5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00000000010e7e2b in pkWaitConditionTracked () #2 0x0000000000cf231e in SdRefreshPoolStats () #3 0x0000000000cf5b20 in sdGetPoolStats () #4 0x0000000000f5bfc4 in ssGetNextPool () #5 0x000000000072376b in bfGetNextPool () #6 0x000000000060cd3c in AdmQueryStgPool () #7 0x0000000000554a98 in AdmCommandLocal () #8 0x00000000005561c9 in admCommand () #9 0x0000000000d59652 in SmAdminCommandThread () #10 0x00000000010e7452 in StartThread () #11 0x00000037c98079d1 in start_thread () from /lib64/libpthread.so.0 #12 0x00000037c94e8b6d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7feae1fcc700 (LWP 26206)): This APAR will address both the poor performance of these commands and the hang of other activity that may still occur since the commands may always be somewhat longer running. Tivoli Storage Manager Versions Affected: 7.1.3 and above Initial Impact: Medium Additional Keywords: TSM IBM Spectrum Protect
Local fix
If attempting to process through the removal of containers at the stgpool or stgpooldir levels by marking all extents damaged then auditing to remove damaged, process the containers one at a time using audit with markdamaged followed by audit with removedamaged.
Problem summary
**************************************************************** * USERS AFFECTED: * * All IBM Spectrum Protect server users. * **************************************************************** * PROBLEM DESCRIPTION: * * See error description. * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in level 8.1.1. Note that this is * * subject to change at the discretion of IBM. * ****************************************************************
Problem conclusion
This problem was fixed. Affected platforms for reported release: AIX, HP-UX, Solaris, Linux, and Windows. Platforms fixed: AIX, Linux, and Windows.
Temporary fix
Comments
APAR Information
APAR number
IT16646
Reported component name
TSM SERVER
Reported component ID
5698ISMSV
Reported release
71A
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-08-17
Closed date
2016-12-06
Last modified date
2016-12-06
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
TSM SERVER
Fixed component ID
5698ISMSV
Applicable component levels
[{"Business Unit":{"code":"BU029","label":"Software"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.1.3"}]
Document Information
Modified date:
01 September 2023