IT26834: SERVER CRASH IN SDWRITECONTAINER FOLLOWING ANR3660E LISTING A CONTAINER MISSING FROM THE FILESYSTEM

8.1.7.000-IBM-SPSRV-Linuxppc64le
8.1.7.000-IBM-SPCMS-WindowsX64
8.1.7.000-IBM-SPCMS-WindowsI32
8.1.7.000-IBM-SPCMS-Linuxx86_64
8.1.7.000-IBM-SPOC-WindowsX64
8.1.7.000-IBM-SPOC-Linuxx86_64
8.1.7.000-IBM-SPOC-Linuxs390x
8.1.7.000-IBM-SPOC-LinuxPPC64le
8.1.7.000-IBM-SPOC-AIX
8.1.7.000-IBM-SPSRV-WindowsX64
8.1.7.000-IBM-SPSRV-Linuxx86_64
8.1.7.000-IBM-SPSRV-Linuxs390x
8.1.7.000-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 7 (V8.1.7) Downloads

APAR status

• Closed as program error.

Error description

• Error Description:
  
  The Spectrum Protect server can crash after encountering a
  container which appears missing at the operating system level.
  
  Customer/L2 Diagnostics (If Applicable)
  
  dsmffdc.log will show :
  
  [10-27-2018 00:40:33.601][ FFDC_GENERAL_SERVER_ERROR ]:
  (psfile.c:2786) Error (platform specific) 2 opening file
  /filesystem/01/02/000000000000003.dcf.
  
  The activity will have a corresponding entry :
  
  10/27/2018 00:40:33 ANR3660E An unexpected error occured while
  opening or writing to the container. Container
  /filesystem/01/02/000000000000003.dcf  in stgpool YOURPOOL has
  been marked as UNAVAILABLE and should be audited to validate
  accessibility and content.
  
  The stack will show similar entries as follows (this example is
  taken from Linux) :
  
  #0 pkDioWrite (dioHandleP=0x0, size=46236, bufP=0x7f1c44d72000
  "\024") at psfile.c:3002
  #1 0x0000000000df01b3 in SdWriteContainer (cntrP=0x7f1c44041bb8,
  bufP=0x7f1c44d72000 "\024", bufLen=46236) at sdio.c:1582
  #2 0x0000000000df0acd in SdWrite (txnP=0x7f1c300f8298,
  chunkP=0x7f1b3a21e658, type=SdContainerTypeDedup,
  cntrPP=0x7f1c302d2010, bufP=<optimized out>, bufLen=135406,
  compressBufP=0x7f1c44d72000 "\024", compressBufSize=4112078) at
  sdio.c:2064
  #3 0x0000000000df21a2 in SdAsyncWrite
  (containerWorkP=0x7f1c301dc4a8) at sdio.c:3328
  #4 0x0000000000e0b3af in AsyncWriteThread (arg1P=<optimized
  out>, arg2P=0xb49c) at sdprodcon.c:5041
  #5 0x00000000011ca6dd in PcConsumerThread (argP=<optimized out>)
  at prodcons.c:653
  #6 0x0000000001259864 in StartThread (startInfoP=0x0) at
  pkthread.c:4026
  #7 0x00007f1d90366dd5 in start_thread () from
  /lib64/libpthread.so.0
  #8 0x00007f1d8bfe3b3d in clone () from /lib64/libc.so.6
  
  The error is coming from the operating system and means :
  
  #define ENOENT   2 /* No such file or directory */
  
  The reason why the container file is suddenly missing requires
  additional investigation depending on end storage used, but the
  code will be amended to handle this scenario without crashing.
  
  In this particular example, the end storage is EMC Isilon using
  NFSv3 protocol and the mounts were managed by dsmisi 3rd party
  software.
  
  IBM Spectrum Protect Versions Affected:
  7.1.3.0 and above, 8.1.0.0 and above, on all platforms.
  
  Initial Impact: Low|Medium|High
  High
  
  Additional Keywords:
  tsm, abend, crashed, core, TS001533706
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 8.1.7. Note that this is      *
  * subject to change at the discretion of IBM.                  *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels