IC76171: TIVOLI STORAGE MANAGER SERVER CAN CRASH WHEN USING MULTIPLE REMO TE NETWORK LIBRARIES SIMULTANEOUSLY

IBM Tivoli Storage Manager V5.5 Fix Pack 6 (5.5.6) Server Downloads

APAR status

• Closed as program error.

Error description

• The Tivoli Storage Manager Server can crash when using multiple
  unique network attached libraries (three or more libraries) at
  the same time.  A network attached library is a library
  connected directly to an NDMP filer appliance for use by NDMP
  backup operations.
  .
  For example, an administrator issues the following BACKUP NODE
  commands simultaneously to three different filers that use
  three different remote(network) libraries:
  .
    BACKUP NODE filer1 /vfs_1 mode=full toc=preferred
    BACKUP NODE filer2 /vfs_2 mode=full toc=preferred
    BACKUP NODE filer3 /vfs_3 mode=full toc=preferred
  .
  This can cause a crash.
  .
  This behavior has also been witnessed during other remote
  library activity, like during remote library initialization
  and remote library label libvol operations.
  .
  Customer/L2 Diagnostics:
  Use a debugger to extract the failing call-chain from the
  coredump generated by the crash.  The following known failing
  call-chains have been associated with this problem(on Linux):
  .
    ps_analyze_io_results()
    dd_scsi_cmd()
    ds_lb_ioctl()
    dd_ioctl()
    TsmRemoteIoctl()
    PvrRemoteAutoIoctl()
    PvrLibAutoIoctl()
    ExecuteAutoOp()
    MoveVolume()
    MountPrivateVolume()
    ScsiMountVolume()
    MmsMountVolume()
    NasOpen()
    AgentThread()
    StartThread()
  .
    ps_analyze_io_results()
    dd_scsi_cmd()
    ds_lb_ioctl()
    dd_ioctl()
    TsmRemoteIoctl()
    PvrRemoteAutoIoctl()
    PvrLibAutoIoctl()
    ExecuteAutoOp()
    AuditSlots()
    ValidateSlotInfo()
    ScsiLabelVolume()
    MmsLtsLabelVolume()
    LabelThread()
    StartThread()
  .
    ps_analyze_io_results()
    dd_scsi_cmd()
    ds_lb_ioctl
    ds_lb_open()
    TsmRemoteOpen()
    PvrRemoteOpenAuto()
    PvrOpenAutoDevice()
    OpenAutoDevice()
    MmsAcquireAuto()
    BuildConfig()
    PerformInit()
    InitThread()
    StartThread()
  .
  A server PVR trace can show the same SCSI ID for the multiple
  unique remote libraries prior to the crash, for example:
  .
    01:02:03.066 ￭00001￭pvrremot.c￭1052￭PvrRemoteOpenAuto:
      Opening the remote library FILER1_LIBRARY(mc0).
    01:02:03.107 ￭00001￭pvrremot.c￭1110￭PvrRemoteOpenAuto:
      Obtained SCSI id is 126 and LUN id is 18.
    01:02:03.433 ￭00002￭pvrremot.c￭1052￭PvrRemoteOpenAuto:
      Opening the remote library FILER2_LIBRARY(mc0).
    01:02:03.434 ￭00002￭pvrremot.c￭1110￭PvrRemoteOpenAuto:
      Obtained SCSI id is 126 and LUN id is 18.
    01:02:03.579 ￭00003￭pvrremot.c￭1052￭PvrRemoteOpenAuto:
      Opening the remote library FILER3_LIBRARY(mc0).
    01:02:03.600 ￭00003￭pvrremot.c￭1110￭PvrRemoteOpenAuto:
      Obtained SCSI id is 126 and LUN id is 15.
  .
  Initial Impact:
  High
  .
  Tivoli Storage Manager Versions Affected:
  All Tivoli Storage Manager Servers using remote libraries
  controlled by the Tivoli Storage Manager pass-through
  device driver.
  .
  Additional Keywords:
  TSM CRASH ABEND ABORT ABORTED CRASH NDMP NAS NASDD
  PS_ANALYZE_IO_RESULTS DD_SCSI_CMD BACKUP NODE FILER
  .
  

Local fix

• Manually serialize the library activity by only using
  a single remote library operation at a time, if possible.
  

Problem summary

• ****************************************************************
  * USERS AFFECTED: All Tivoli Storage Manager Server users      *
  *                 with multiple Tape Libraries configured      *
  *                 for NAS backup.                              *
  ****************************************************************
  * PROBLEM DESCRIPTION: See ERROR DESCRIPTION.                  *
  ****************************************************************
  * RECOMMENDATION: Apply fixing level when available. This      *
  *                 problem is currently projected to be fixed   *
  *                 in levels 5.4.7, 5.5.6, 6.1.6, and 6.2.3.    *
  *                 Note that this is subject to change at the   *
  *                 discretion of IBM.                           *
  ****************************************************************
  See ERROR DESCRIPTION.
  Affected Platforms: AIX, HP-UX, Linux, Sun Solaris, and Windows.
  

Problem conclusion

• The described problem has been resolved.
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels

• R54A PSY

     UP

• R54H PSY

     UP

• R54L PSY

     UP

• R54S PSY

     UP

• R54W PSY

     UP

• R55A PSY

     UP

• R55H PSY

     UP

• R55L PSY

     UP

• R55S PSY

     UP

• R55W PSY

     UP

• R61A PSY

     UP

• R61H PSY

     UP

• R61L PSY

     UP

• R61S PSY

     UP

• R61W PSY

     UP

• R62A PSY

     UP

• R62H PSY

     UP

• R62L PSY

     UP

• R62S PSY

     UP

• R62W PSY

     UP