IT37116: NDMP BACKUP TO A CONTAINER STGPOOL FAILS WITH ANR9999D_2352532847 AND THE SERVER CRASHES WITH A CORE DUMP

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• After a NDMP backup to container failed with ANR9999D_2352532847
  the server crashes with a core dump.
  
  The ACTLOG might show the following entries:
  ANR1104E NAS Backup to Server Storage process 74 terminated -
  NAS datamover xxxx reported NDMP session errors.
  ANR0171I tbcli.c(2295): Error detected on 385:12, database in
  evaluation mode.
  ANR0162W Supplemental database diagnostic information:
  -1:40003:-1224 ([IBM][CLI Driver] SQL1224N  The database manager
  is not able to accept new requests, has terminated all requests
  in progress, or has terminated the specified request because of
  an error or a forced interrupt.  SQLSTATE=55032).
  ANR0102E tbcli.c(2596): Error 4522 inserting row in table
  "NDMP.Nodes(TEMP)".
  ANR9999D_2352532847 tbDestroyTemp(tbtbl.c:8326) Thread<xxx>:
  Insert cache flush failure, rc=4522
  ANR9999D Thread<xxx>      0x132fbf2
  /opt/tivoli/tsm/server/bin/dsmserv(OutDiagToCons+0xa2)
  ANR9999D Thread<xxx>      0x133788c
  /opt/tivoli/tsm/server/bin/dsmserv(outDiagfExt+0x1fc)
  ANR9999D Thread<xxx>      0xd82870
  /opt/tivoli/tsm/server/bin/dsmserv(tbDestroyTemp+0x190)
  ANR9999D Thread<xxx>      0x12a8216
  /opt/tivoli/tsm/server/bin/dsmserv(tocBuildCancel+0x146)
  ANR9999D Thread<xxx>      0x8b6868
  /opt/tivoli/tsm/server/bin/dsmserv(EndRemoteProc+0x108)
  ANR9999D Thread<xxx>      0x8b9910
  /opt/tivoli/tsm/server/bin/dsmserv(AfStoreNativeThread+0- x490)
  
  getcoreinfo.txt shows:
  Program terminated with signal 11, Segmentation fault.
  #0  0x00007f96c07d2d00 in pthread_mutex_lock () from
  /lib64/libpthread.so.0
  
  ================================================================
  =========
  
  #0  0x00007f96c07d2d00 in pthread_mutex_lock () from
  /lib64/libpthread.so.0
  #1  0x00000000013d0735 in pkAcquireMutexTracked
  (mutexP=mutexP@entry=0x7f94dc2192b0, file=file@entry=0x17fd060
  "sdcreate.c", line=line@entry=2869) at pkmon.c:604
  #2  0x0000000000ed7eb7 in SdStore
  (sessP=sessP@entry=0x7f94dc219080,
  srcFunc=srcFunc@entry=0x7f96aa8fe7f0 <NasRecvNextData>,
  contextP=contextP@entry=0x7f94f1b39000,
  bfId=bfId@entry=427703651, estSize=estSize@entry=54251673600) at
  sdcreate.c:2869
  #3  0x0000000000ed99fb in sdCreate (txnId=<optimized out>,
  poolId=<optimized out>, bfId=bfId@entry=427703651,
  ck1=ck1@entry=689, ck2=ck2@entry=8,
  estSize=estSize@entry=54251673600, srcFunc=0x7f96aa8fe7f0
  <NasRecvNextData>, contextP=0x7f94f1b39000,
  actSizeP=0x7f93c229f8a8, rptSizeP=0x7f93c229fa28, isPaMb=False)
  at sdcreate.c:1793
  #4  0x000000000078c67d in CreateBitfileEx
  (sessP=sessP@entry=0x7f94dc14eea0, txnId=0x7f9410024910,
  bfId=427703651, estSize=estSize@entry=10485760000,
  estBitfileSize=estBitfileSize@entry=54251673600,
  ck1=ck1@entry=689, ck2=8, poolName=0x7f94f1b39278
  "DIRECTORYPOOL", mountWaitMode=bfWaitMount,
  sourceFunc=0x7f96aa8fe7f0 <NasRecvNextData>,
  contextP=0x7f94f1b39000, aggregateState=admNoAggregate,
  forceAggregation=False, rptSizeP=0x7f93c229fa28,
  aggrSizeLimit=True, isPaMb=False) at bfcreate.c:10210
  #5  0x000000000078e29c in bfCreateEx (sessHandle=<optimized
  out>, txnId=0x7f920021c2b0, bfId=<optimized out>,
  estSize=<optimized out>, estBitfileSize=54251673600, ck1=689,
  ck2=8, poolName=0x7f94f1b39278 "DIRECTORYPOOL",
  mountWaitMode=bfWaitMount, sourceFunc=0x7f96aa8fe7f0
  <NasRecvNextData>, contextP=0x7f94f1b39000,
  aggregateState=admNoAggregate, forceAggregation=False,
  isFragmentedP=0x0, rptSizeP=0x0, isPaMb=False) at
  bfcreate.c:2753
  #6  0x00000000007ab185 in bfNASCreate (argsP=0x7f94f1b39248,
  sourceFuncP=0x7f96aa8fe7f0 <NasRecvNextData>,
  contextP=0x7f94f1b39000) at bfremote.c:2526
  #7  0x00007f96aa8fe5b0 in moverAcceptConnection
  (cookie=0x7f94f1b39000, fd=62, mode=<optimized out>) at
  ndserver.c:1867
  #8  0x00007f96aa9061f3 in ndmpdSelect
  (fileHandlerList=<optimized out>, block=block@entry=0,
  classMask=classMask@entry=4294967295) at
  ../../unix/ndmpconn.c:1383
  #9  0x00007f96aa8ffed2 in connectionHandler (argP=<optimized
  out>) at ndserver.c:697
  #10 0x00000000013ce7b0 in StartThread (startInfoP=<optimized
  out>) at pkthread.c:4282
  #11 0x00007f96c07d0ea5 in start_thread () from
  /lib64/libpthread.so.0
  #12 0x00007f96bce308cd in clone () from /lib64/libc.so.6
  
  
  IBM Spectrum Protect Server Version Affected:
  Version 8.1.2.000 and above on all supported platforms
  
  Initial Impact:
  Medium
  
  Additional Keywords:
  TSM server Spectrum Protect TS005257797 core crash NDMP
  container
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users of NDMP backup to      *
  * container pools.                                             *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed. Affected platforms for reported release:
  AIX, Linux, and Windows. Platforms fixed: AIX, Linux, Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP