IT26003: ANR0548W RESTORING/RETRIEVING DATA FROM CONTAINER POOL

7.1.10.000-TIV-TSMSRV-WIN
7.1.10.000-TIV-TSMSRV-SolarisSPARC
7.1.10.000-TIV-TSMSRV-Linuxx86_64
7.1.10.000-TIV-TSMSRV-Linuxs390x
7.1.10.000-TIV-TSMSRV-Linuxppc64
7.1.10.000-TIV-TSMSRV-HP-UX
7.1.10.000-TIV-TSMSRV-AIX
8.1.1.400-IBM-SPSRV-WindowsX64
8.1.1.400-IBM-SPSRV-Linuxx86_64
8.1.1.400-IBM-SPSRV-Linuxs390x
8.1.1.400-IBM-SPSRV-Linuxppc64le
8.1.1.400-IBM-SPSRV-AIX
8.1.7.000-IBM-SPSRV-Linuxppc64le
8.1.7.000-IBM-SPCMS-WindowsX64
8.1.7.000-IBM-SPCMS-WindowsI32
8.1.7.000-IBM-SPCMS-Linuxx86_64
8.1.7.000-IBM-SPOC-WindowsX64
8.1.7.000-IBM-SPOC-Linuxx86_64
8.1.7.000-IBM-SPOC-Linuxs390x
8.1.7.000-IBM-SPOC-LinuxPPC64le
8.1.7.000-IBM-SPOC-AIX
8.1.7.000-IBM-SPSRV-WindowsX64
8.1.7.000-IBM-SPSRV-Linuxx86_64
8.1.7.000-IBM-SPSRV-Linuxs390x
8.1.7.000-IBM-SPSRV-AIX
7.1.9.200-TIV-TSMSRV-WIN
7.1.9.200-TIV-TSMSRV-SolarisSPARC
8.1.1.300-IBM-SPSRV-AIX
8.1.1.300-IBM-SPSRV-Linuxppc64le
8.1.1.300-IBM-SPSRV-Linuxs390x
8.1.1.300-IBM-SPSRV-Linuxx86_64
8.1.1.300-IBM-SPSRV-WindowsX64
7.1.9.100-TIV-TSMSRV-AIX
7.1.9.100-TIV-TSMSRV-HP-UX
7.1.9.100-TIV-TSMSRV-Linuxppc64
7.1.9.100-TIV-TSMSRV-Linuxs390x
7.1.9.100-TIV-TSMSRV-Linuxx86_64
7.1.9.100-TIV-TSMSRV-SolarisSPARC
7.1.9.100-TIV-TSMSRV-WIN
7.1.9.200-TIV-TSMSRV-AIX
7.1.9.200-TIV-TSMSRV-HP-UX
7.1.9.200-TIV-TSMSRV-Linuxppc64
7.1.9.200-TIV-TSMSRV-Linuxs390x
7.1.9.200-TIV-TSMSRV-Linuxx86_64
IBM Spectrum Protect Server V8.1.1.X interim fix downloads
IBM Spectrum Protect Server V7.1.7.X interim fix downloads
IBM Spectrum Protect Server V7.1.9.X interim fix downloads
IBM Spectrum Protect Server V8.1.6.X interim fix downloads
IBM Spectrum Protect Server V8.1 Fix Pack 7 (V8.1.7) Downloads
IBM Spectrum Protect Server V7.1 Fix Pack 10 (7.1.10.000) Downloads

APAR status

• Closed as program error.

Error description

• A Spectrum Protect Backup-Archive client may fail to restore
  or retrieve an object from a directory container pool.
  
  The client logs the following errors :
  
  ANS1314E File data currently unavailable on server
  ANS5042E File data is currently unavailable on the
   IBM Spectrum Protect server
  
  The server logs the ANR0548W message.
  
  This also affects VM backups as the VM control file needs
  to be restored during an IFINCR backup. In that case, the
  server logs the following error :
  
  ANR0548W Retrieve or restore failed for session 123 for node
  MYNODE (TDP VMware) processing file space
  \VMFULL-VMNAME (fsId 456) for file
  \VMNAME\xxx\Hard Disk n\xxx\MBLK000000xx.CTL
  stored as Backup - error detected.
  
  This issue is a result of a rare window in which a container
  reference in the database can be removed prematurely, more
  likely to occur during heavy workload and poor server
  performance.
  Customer/L2 Diagnostics :
  A Spectrum Protect server trace (SD) will show the server is
  unable to find a container :
  
  <timestamp> [nnn][sdcntr.c][2828][SdGetContainerDesc]:
  Container 789 not found in database
  ...
  <timestamp> [nnn][sdrtrv.c][3670][OpenContainer]:
   Exit, rc 1116, cntrId 789
  ...
  <timestamp> [nnn][sdcntr.c][3753][SdReestablishContainer]:
  Exit: rc 1101
  
  Logon with the server instance userid and run the following
  commands from a shell to determine the number of missing
  containers :
  
  db2 connect to tsmdb1
  db2 set schema tsmdb1
  db2 "select count(distinct cntrid) from sd_all_chunks sdac where
   not exists (select 1 from sd_containers sdcn
   where sdcn.cntrid=sdac.cntrid) for read only with ur"
  A count > 0 indicates there is one or more missing containers.
  
  Initial Impact: High
  
  Additional Keywords:
  SDDA GRC_ITEM_NOT_FOUND GRC_BITFILE_NOT_FOUND TS001086972
  
  Versions Affected:
  - For IBM Spectrum Protect server V7.1,
     versions 7.1.3 and above on all platforms are affected.
  - For IBM Spectrum Protect server V8.1,
     versions below 8.1.3 on all platforms are affected.
  

Local fix

• Contact support for data recovery assistance.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 7.1.9.200, 7.1.10,           *
  * 8.1.6.100, and 8.1.7. Note that this is subject to change at *
  * the discretion of IBM.                                       *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Solaris, Linux,
  and Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R71A PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71W PSY

     UP

• R81A PSY

     UP

• R81L PSY

     UP

• R81W PSY

     UP