IT13393: TIVOLI STORAGE MANAGER API INITIALIZATION CRASH WHEN MULTIPLE THREADS ACCESS THE PASSWORD FILE SIMULTANEOUSLY

IBM Spectrum Protectâ„¢ Client 7.1.6 Downloads and READMEs

APAR status

• Closed as program error.

Error description

• When an application (such as Content Manager Ondemand in this
  case) uses the Tivoli Storage Manager API, and has an extremely
  heavy load (5000+ sessions per minute), a crash can occur during
  the initialization of the API when multiple threads are
  simultaneously accessing the Password file.
  The errorlog may (or may not) contain many messages such as:
    ANS2261E TSM client failed to lock options file.
  and when the crash occurs, may (or may not) record the error:
    ANS0361I DIAG: Block header invalid, possible memory
    overwrite. Addr 0x7f8abd5f6bf0, File psPasswordFile.cpp,
    Line 1661
  This problem has only been seen in rare cases with a crash on
  arssockd if the memory used for all the threads of the multiple
  sessions exceed 1GB.
  .
  Tivoli Storage Manager Versions Affected:  Tivoli Storage
  Manager Client version 7.1.0 and greater
  .
  Customer/L2 Diagnostics
  The stack for the crash would look like:
  (gdb) bt
  #0 0x0000003086032625 in raise () from /lib64/libc.so.6
  #1 0x0000003086033e05 in abort () from /lib64/libc.so.6
  #2 0x0000003086070537 in __libc_message () from /lib64/libc.so.6
  #3 0x0000003086075e66 in malloc_printerr () from
  /lib64/libc.so.6
  #4 0x00007f8b228b7d43 in dsmFree(void*, char*, unsigned int) ()
  from /usr/lib64/libApiTSM64.so
  #5 0x00007f8b22da7d7b in
  psPasswordFile::setPasswordFile(clientOptions const*) () from
  /usr/lib64/libApiTSM64.so
  #6 0x00007f8b227d2524 in ApiInitEx(unsigned int*,
  tsmInitExIn_t*, tsmInitExOut_t*, int) () from
  /usr/lib64/libApiTSM64.so
  #7 0x00007f8b227d3a3e in tsmInitEx () from
  /usr/lib64/libApiTSM64.so
  #8 0x00007f8b227d3cce in dsmInitEx () from
  /usr/lib64/libApiTSM64.so
  #9 0x00007f8b23340571 in ArcSMSP_BuildSMS(_ArcSMType,
  _ArcSMDelimType, char const*, char const*, char const*,
  SMSError&, char const*, _ArcSMSStartupHandle*) ()
  from /opt/ibm/ondemand/V9.5/bin/exits/arssmtsm
  #10 0x00000000007b01eb in ArcSMS_Init ()
  #11 0x000000000056617e in ?? ()
  #12 0x000000000056685c in ?? ()
  #13 0x00000000005653a3 in ?? ()
  #14 0x0000000000575761 in ArcCSSM_DocGet ()
  #15 0x0000000000538c38 in ArcCSP_DocGet ()
  #16 0x00000000005a19e9 in ArcXPORT_ServerRequestProcess ()
  #17 0x00000000004f3e6c in ?? ()
  #18 0x00000000004f5dac in ?? ()
  #19 0x00000030864079d1 in start_thread () from
  /lib64/libpthread.so.0
  #20 0x00000030860e89dd in clone () from /lib64/libc.so.6
  .
  Additional Keywords: TSM Spectrum Protect
  

Local fix

• Specify the TESTFLAG APIINITMUTEX in the user options file
  (default is dsm.opt) that is utilized by the API application.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * Tivoli Storage Manager for API version 7.1 running on Unix   *
  * and Linux                                                    *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See ERROR DESCRIPTION                                        *
  ****************************************************************
  * RECOMMENDATION:                                              *
  * Apply fixing level when available. This problem is currently *
  * projected to be fixed in level 7.1.6.                        *
  * Note that until these levels are available, this information *
  * is subject to change at the discretion of IBM.               *
  ****************************************************************
  *
  

Problem conclusion

• The fix added new synchronization point which prevents freeing
  an object in parallel in different threads.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Modules/Macros

• api
  

Fix information

• Fixed component name

  TSM CLIENT

• Fixed component ID

  5698ISMCL

Applicable component levels

• R71A PSY

     UP

• R71L PSY

     UP

• R71S PSY

     UP

• R71H PSY

     UP

• R71M PSY

     UP