IT38178: Server crashes while writing instrumentation data to file

8.1.12.200-IBM-SPSRV-WindowsX64
8.1.12.200-IBM-SPSRV-Linuxx86_64
8.1.12.200-IBM-SPSRV-Linuxs390x
8.1.12.200-IBM-SPSRV-Linuxppc64le
8.1.12.200-IBM-SPSRV-AIX
IBM Spectrum Protect Server V8.1 Fix Pack 13 (V8.1.13) Downloads
IBM Spectrum Protect Server V8.1.12.X interim fix downloads

APAR status

• Closed as program error.

Error description

• IBM Spectrum Protect server crashes with an ACCESS_VIOLATION and
  the stack below when printing out instr data. The
  alwaysonservermon feature will run server instrumentation
  tracing on a periodic basis as part of it's diagnostic
  collection.
  
  
  When seen on a Windows server the dsmserv.dmp file shows the
  following:
  
  Crash dump file:
  
  C:\Program Files\Tivoli\TSM\server\dsmsvc.dmp
  
  Exception code:
  
  C0000005 (ACCESS_VIOLATION) at 000000007FCA2C76
  
  Call stack:
  
  Address          Parameters
                                F()+Offset, File(Line#)
  ---------------------------------------------------------------
  -------------------------------------------------------
  000000007FCA2C76 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000000000000000) Unavailable, Unavailable
  00007FFF7FC94341 (0x00000086689FC910 0x00000086689FC910
  0x00000000FFFFFFFF 0x0000000000000001) fcloseall()+531,
  Unavailable
  00007FFF7FC94844 (0x0000008668000000 0x00000086689FC910
  0x0000000000000000 0x00000086689FC938) fcloseall()+A34,
  Unavailable
  00007FFF7FC94AEB (0x00000086689FCEC8 0x00000086689FC970
  0x0000024A57A6D000 0x0000000000007D00) fcloseall()+CDB,
  Unavailable
  00007FFF7FC97B45 (0xFFFFFFFFFFFFFFFF 0x0000024A5DA22540
  0x00000086689FCFF0 0x0000000000000001)
  _stdio_common_vsprintf()+105, Unavailable
  00007FFF43948AD1 (0x0000000000000000 0x00000086689FCEC8
  0x502C024A4F16BCD0 0x502C024A4F16BCD0) sprintf()+51, Unavailable
  
  00007FFF44673C71 (0x00000000000007DA 0x00007FFF4395B3E0
  0x0000024A044F15D0 0x0000000000000008) PrepDispArgs()+A21,
  Unavailable
  00007FFF44672CD0 (0x0000000000000000 0x0000000000000000
  0x0000000000FFF600 0x00007FFF469114E0) outPrintfTrFL()+50,
  Unavailable
  00007FFF44692093 (0x0000000000000001 0x0000024A61F9F3F0
  0xFFFFFFFFFFFFFFFF 0x0000024A66B08330) instrumentFileShow()+923,
  Unavailable
  00007FFF43ACC215 (0x0000024A61F9F3F0 0x00000086689FED71
  0x00007FFF44A80440 0x00007FFF44A80440) AdmInstrumentEnd()+215,
  Unavailable
  00007FFF439FC500 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000024A61F9F3F0) AdmCommandLocal()+760,
  Unavailable
  00007FFF439FB6D3 (0x000000000000013A 0x00007FFF448BD060
  0x0000024A6325E280 0x0000024A62C2B550) admCommand()+1533,
  Unavailable
  00007FFF4469CAA5 (0x0000024A000001D6 0x0000024A6325E280
  0x0000024A61F9F3F0 0x00007FFF4481F07C) RunCommand()+195,
  Unavailable
  00007FFF4469DDEC (0x0000024A68C27900 0x00007FFF4469D910
  0x0000000000000000 0x0000024A68C27900)
  ServerMonTwentyMinThread()+4DC, Unavailable
  00007FFF4394D2D3 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000000000000000) startThread()+5B3,
  Unavailable
  00007FFF7FC9FB80 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000000000000000) o__realloc_base()+60,
  Unavailable
  00007FFF82A984D4 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000000000000000) BaseThreadInitThunk()+14,
  Unavailable
  00007FFF82BD1791 (0x0000000000000000 0x0000000000000000
  0x0000000000000000 0x0000000000000000) RtlUserThreadStart()+21,
  Unavailable
  
  Exception handler complete. RC: 0
  
  When looking at the stack from windbgx64 it shows:
  
  ucrtbase!strnlen+0x46
  ucrtbase!fcloseall+0x531
  ucrtbase!fcloseall+0xa34
  ucrtbase!fcloseall+0xcdb
  ucrtbase!_stdio_common_vsprintf+0x105
  adsmdll!sprintf+0x51
  adsmdll!PrepDispArgs+0xa21
  adsmdll!outPrintfTrFL+0x50
  adsmdll!instrumentFileShow+0x923
  adsmdll!AdmInstrumentEnd+0x215
  adsmdll!AdmCommandLocal+0x760
  adsmdll!admCommand+0x1533
  adsmdll!RunCommand+0x195
  adsmdll!ServerMonTwentyMinThread+0x4dc
  adsmdll!startThread+0x5b3
  ucrtbase!o__realloc_base+0x60
  kernel32!BaseThreadInitThunk+0x14
  ntdll!RtlUserThreadStart+0x21
  
  This APAR only applies if "instrumentFileShow" is seen in the
  failing call stack.
  
  IBM Spectrum Protect Versions Affected: 8.1.11.000 and above on
  all platforms
  
  Additional Keywords: TSM TS004635382
  
  | MDVREGR 8.1.11.0-TIV_5698MSV |
  

Local fix

• Disable instr tracing in alwaysonservermon by doing:
  
       edit $installdir/bin/servermon/commands.ini and
  $instancedir/srvmon/commands.ini and add a '#' in front of the
  two instrumentation commands
  
      #INSTR END
      #INSTR BEGIN
  
  
  Additionally don't run server instr tracing manually.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Spectrum Protect server users.                       *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in levels 8.1.12.200 and 8.1.13. Note  *
  * that this is subject to change at the discretion of IBM.     *
  ****************************************************************
  

Problem conclusion

• This problem was fixed.
  Affected platforms for reported release:  AIX, Linux, and
  Windows.
  Platforms fixed:  AIX, Linux, and Windows.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R81A PSY

     UP