Direct links to fixes
APAR status
Closed as program error.
Error description
In a large LAN-free environment, tape mounts become very slow and all LAN-free backups can not be done in expected time. The environment where this problem happened had several libraries with more than 100 drives each, 90 LAN-free storage agents, and several thousands of tape paths defined. This IBM Spectrum Protect select can be used to see the increased "idle" wait time for the storage agent session: select start_time, end_time, number, idle, BYTES_PROTECTED, BYTES, activity from summary where activity='BACKUP' and entity='<NODE NAME>' order by start_time In servermon, the instrumentation trace summary shows a lot of time spent in Tm Lock Wait and Acquire Latch: TOTAL SERVER SUMMARY Operation Count Tottime Avgtime Maxtime InstTput RealTput Total KB --------------------------------------------------------------- Disk Read 4393 5.726 0.001 0.030 191050.0 925.0 1094016 Some lines removed for easier reading Tm Lock Wait 4016 133917.326 33.346 1160.803 Acquire Latch 43337 103275.002 2.383 260.849 Acquire XLatch 466394 0.040 0.000 0.000 lines removed The locking / lactching is mainly affecting storage agents server type sessions, like example below: Thread 88547 psSessionThread parent=323 2023-12-08T12:41:10.487-->2023-12-08T13:00:53.244 PORT:10.xx.yy.zz:26433,SESSION:60669,SESSION_TYPE:SERVER Operation Count Tottime Avgtime Mintime Maxtime InstTput Total KB --------------------------------------------------------------- DB2 Fetch Exec 55 0.283 0.005 0.000 0.057 lines removed Tm Lock Wait 16 485.196 30.325 0.176 127.312 Acquire Latch 23 674.928 29.345 0.074 239.926 Unknown 20.718 --------------------------------------------------------------- Total 1182.756 The servermon files *10min-show.xml may not all be collected, so it may be necessary to collect SHOW THREAD output manually to see threads stack trace details. In SHOW THREADS outputs, the affected storage agent threads are mainly waiting on some devclass latch, like in pvrIsFileDevClass or PvrGetDevClassLatches, for example: Thread 88928, Parent 323: psSessionThread, Storage 2801376, AllocCnt 670 HighWaterAmt 3839840 tid=6cf17, ptid=e143, det=1, zomb=0, join=0, result=0, sess=0, procToken=0, sessToken=60930 Stack trace: _cond_wait_global _cond_wait pthread_cond_wait pkWaitConditionTracked AcquireLatchSpecific pvrIsFileDevClass AsGetPoolStatus ssGetNextPool IPRA.$DoSendStgPoolInfo IPRA.$DoSendConfigInfo SmHandleAgentVerb SmNodeSession IPRA.$DoDbServer smExecuteSession psSessionThread StartThread Awaiting cond latchP->sFree (0x11121db00), using mutex PVRV->mutex (0x1146aa0f0), at latch.c(275) And/or LAN-free sessions are waiting on mutex NAV->Mutex like in naGetNextPath for example: Thread 90693, Parent 323: psSessionThread, Storage 2247888, AllocCnt 169 HighWaterAmt 2247888 tid=33d0b, ptid=e143, det=1, zomb=0, join=0, result=0, sess=62104, procToken=0, sessToken=62104 Stack trace: _global_lock_common _mutex_lock pkAcquireMutexTracked naGetNextPath naTestAnyDrivePath pvrTestLanFreePath ssLANFreePathtoPool IPRA.$FormatBackupCopyGroup IPRA.$FormatMgmtClass pmQueryActivePolicySet SmDoPSQry SmNodeSession IPRA.$HandleNodeSession smExecuteSession psSessionThread StartThread Acquiring mutex NAV->mutex (0x113d07950) at napthcmd.c(2400) And/or also waiting on lock DoSendStgPoolInfo->ssGetNextPool->A sGetPoolStatus->tmLockTracked->WaitForLocklike for example: Thread 88483, Parent 323: psSessionThread, Storage 3052704, AllocCnt 762 HighWaterAmt 3976944 tid=5ce55, ptid=e143, det=1, zomb=0, join=0, result=0, sess=0, procToken=0, sessToken=60642 Stack trace: _cond_wait_global _cond_wait pthread_cond_wait pkWaitConditionTracked IPRA.$WaitForLock tmLockTracked AsLockVolIdTracked AsGetPoolStatus ssGetNextPool IPRA.$DoSendStgPoolInfo IPRA.$DoSendConfigInfo SmHandleAgentVerb SmNodeSession IPRA.$DoDbServer smExecuteSession psSessionThread StartThread Awaiting cond waitP->waiting (0x2501ae920), using mutex TMV->mutex (0x111678090), at tmlock.c(2539) IBM Spectrum Protect Versions Affected: IBM Spectrum Protect server 8.1 and above on all supported platforms Additional Keywords: TS014874390 TSM lanfree storage
Local fix
Decrease the number of drive paths as well as the backup workload until backup performance meets the expectation.
Problem summary
**************************************************************** * USERS AFFECTED: * * All IBM Storage Protect server users. * **************************************************************** * PROBLEM DESCRIPTION: * * See error description. * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in level 8.1.23. Note that this is * * subject to change at the discretion of IBM * ****************************************************************
Problem conclusion
This problem was fixed. Affected platforms: AIX, Linux, and Windows.
Temporary fix
Comments
APAR Information
APAR number
IT45608
Reported component name
TSM SERVER
Reported component ID
5698ISMSV
Reported release
81A
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2024-03-01
Closed date
2024-05-16
Last modified date
2024-05-16
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
TSM SERVER
Fixed component ID
5698ISMSV
Applicable component levels
[{"Business Unit":{"code":"BU029","label":"Software"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"81A"}]
Document Information
Modified date:
14 June 2024