Direct links to fixes
8.1.15.100-IBM-SPSRV-WindowsX64
8.1.15.100-IBM-SPSRV-Linuxx86_64
8.1.15.100-IBM-SPSRV-Linuxs390x
8.1.15.100-IBM-SPSRV-Linuxppc64le
8.1.15.100-IBM-SPSRV-AIX
8.1.16.000-IBM-SPCMS-WindowsX64
8.1.16.000-IBM-SPCMS-WindowsI32
8.1.16.000-IBM-SPCMS-Linuxx86_64
8.1.16.000-IBM-SPOC-WindowsX64
8.1.16.000-IBM-SPOC-Linuxx86_64
8.1.16.000-IBM-SPOC-Linuxs390x
8.1.16.000-IBM-SPOC-LinuxPPC64le
8.1.16.000-IBM-SPOC-AIX
8.1.16.000-IBM-SPSRV-WindowsX64
8.1.16.000-IBM-SPSRV-Linuxx86_64
8.1.16.000-IBM-SPSRV-Linuxs390x
8.1.16.000-IBM-SPSRV-Linuxppc64le
8.1.16.000-IBM-SPSRV-AIX
8.1.14.200-IBM-SPSRV-WindowsX64
8.1.14.200-IBM-SPSRV-Linuxx86_64
8.1.14.200-IBM-SPSRV-Linuxs390x
8.1.14.200-IBM-SPSRV-Linuxppc64le
8.1.14.200-IBM-SPSRV-AIX
APAR status
Closed as program error.
Error description
This APAR will address long running SQL (select statements) that are used for retention set handling during IBM Spectrum Protect server inventory expiration processing. This APAR only applies if inventory expiration is run against nodes/file spaces that contain data in retention sets. Customer/Support Diagnostics: The long running SQL can be seen as gaps in processing during inventory expiration. The gaps in processing can be observed in the following ways. 1. In the server activity log. The node ANOTHER_NODE, file space /another/filespace, is a specific node/filespace that contains a very large number of objects in retention sets, in the order of hundreds of millions or greater. A gap can be observed before expiration is observed as starting against this files pace. For example: 09/10/2021 14:57:43 ANR0165I Inventory file expiration started processing for node CLIENT_NODE, file space /filespace/name, copygroup BACKUP and object type DIRECTORY. (SESSION: XXXXX, PROCESS: YYY) 09/10/2021 14:57:43 ANR2753I (HOUSEKEEPING):ANR0166I The inventory expiration process is completed for node CLIENT_NODE,/filespace/name , file space /filespace/name, copy group BACKUP, and object type DIRECTORY with the following results: examined 3, retained 0, deleted 3, retried 0, failed 0, and skipped 0 objects. 0 retention bitfiles were deleted from retention pools. (SESSION: XXXXX) ------------->>>>>Gap here for nearly 2.25 hours------------------>>>> 09/10/2021 17:18:24 ANR0165I Inventory file expiration started processing for node ANOTHER_NODE, filespace /another/filespace, copygroup BACKUP and object type FILE. (SESSION: XXXXX, PROCESS: YYY) 2. Gaps in inventory expiration can be observed in a IBM Spectrum Protect server trace with the trace classes "IMEXP RETSIM" enabled. gap1: 09:49:58.876 [1724601][imdmgr.c][5457][ExpireBackupData]: Decrementing 2021-08-26 09:39:10 by 0 days for retExtra and 0 days for retOnly 10:38:44.414 [1724601][imret.c][9060][ImOpenSQLForCopiesInPrimaryPool]:Opened SQL to fetch retained objects with primary copies to delete under nodeId 9, fsId 1 only, rc 0. gap2: 10:54:02.046 [1724601][imret.c][8902][ImGetExpiringRetSets]: Exit 0, any retention set found False, mostInsertTime(1900-00-00 00:00:00), leastDeacTime(2155-255-255 255:255:255) 12:25:32.403 [1724601][imret.c][9270][ImOpenSQLForOrphanRtndObjs]:Opened SQL to fetch orphan retained objects without retention sets under nodeId 9, fsId 1 only, rc 0. 3. An example of the two affected SQL, seen in the servermon data, (*-20min.txt file), the first field is the elapse time in seconds: 2709,EXEC, SELECT IMBK.OBJID, IMBK.BFSIZE, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,458752)>0 THEN 1 END AS ISIMGLLEAD, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,7)>0 THEN 1 END AS ISIMGLMEMB, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,131074)>0 THEN 1 END AS ISIMGLPEER, IMBK.OBJCHGID, IMBK.FLAGS, IMBK.DEACDATE, IMBK.INSDATE, IMBK.STATE, IMBK.RETSET_LOC FROM TSMDB1.BACKUP_OBJECTS IMBK WHERE IMBK.NODEID=? AND IMBK.FSID=? AND IMBK.STATE=4 AND IMBK.RETSET_LOC=17 AND ( IMBK.GROUPTYPE=131072 OR IMBK.GROUPTYPE IS NULL ) OPTIMIZE FOR 2048 ROWS FOR READ ONLY WITH UR --<thread ID> 4865,EXEC, SELECT IMBK.OBJID, IMBK.BFSIZE, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,458752)>0 THEN 1 END AS ISIMGLLEAD, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,7)>0 THEN 1 END AS ISIMGLMEMB, CASE WHEN IMBK.GROUPTYPE IS NOT NULL AND BITAND(IMBK.GROUPTYPE,131074)>0 THEN 1 END AS ISIMGLPEER, IMBK.OBJCHGID, IMBK.FLAGS, IMBK.DEACDATE, IMBK.INSDATE, IMBK.STATE, IMBK.RETSET_LOC FROM TSMDB1.BACKUP_OBJECTS imbk WHERE imbk.NODEID=? AND imbk.FSID=? AND (imbk.GROUPTYPE=131072 OR imbk.GROUPTYPE IS NULL) AND imbk.STATE=4 AND NOT EXISTS ( SELECT 1 FROM TSMDB1.retention_sets rs WHERE rs.NODEID=imbk.NODEID AND rs.FSID=imbk.FSID AND rs.STATE<4 AND imbk.INSDATE <= rs.STARTDATE AND imbk.DEACDATE > rs.STARTDATE ) OPTIMIZE FOR 2048 ROWS FOR READ ONLY WITH UR -- <thread ID> The fix for this APAR will reduce, but not eliminate, the observed gap. IBM Spectrum Protect Versions Affected: v8.1.7 and higher, all platforms. Additional Keywords: delay slow hung hang
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: * * All IBM Spectrum Protect server users. * **************************************************************** * PROBLEM DESCRIPTION: * * See error description. * **************************************************************** * RECOMMENDATION: * * Apply fixing level when available. This problem is currently * * projected to be fixed in levels 8.1.14.200, 8.1.15.100 and * * 8.1.16. Note that this is subject to change at the * * discretion of IBM. * ****************************************************************
Problem conclusion
This problem was fixed. Affected platforms for reported release: AIX, Linux, and Windows. Platforms fixed: AIX, Linux, and Windows.
Temporary fix
Comments
APAR Information
APAR number
IT40249
Reported component name
TSM SERVER
Reported component ID
5698ISMSV
Reported release
81L
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2022-03-11
Closed date
2022-10-24
Last modified date
2022-10-24
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
TSM SERVER
Fixed component ID
5698ISMSV
Applicable component levels
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"81L","Line of Business":{"code":"LOB26","label":"Storage"}}]
Document Information
Modified date:
01 November 2022