IT48644: OPEN SNAP STORE MANAGER (OSSM) PRIMARY AGENT PANIC AFTER UPGRADE TO 8.1.27

APAR status

• Closed as program error.

Error description

• The IBM Storage Protect Open Snap Store Manager (OSSM) primary
  agent can crash upon service startup.
  The service then is respawned continuously and stays in the
  activating status :
  
  ossm.service     Loaded: loaded
  (/etc/systemd/system/ossm.service; enabled;
              preset: disabled)
      Active: activating (auto-restart) (Result: core-dump)
              since <timestamp> CEST; <xx>s ago
     Process: <pid>  ExecStart=/bin/bash -c source
              ~/sqllib/db2profile;
              /opt/tivoli/tsm/ossm/bin/ossmctl primary -c
              /<SPServerInstanceUserHomeDirectory>/ossm/
              config_primary.json (code=dumped, signal=ABRT)
    Main PID: <pid> (code=dumped, signal=ABRT)
         CPU: 611ms
  
  In the OSSM log
  /<SPServerInstanceUserHomeDirectory>/ossm/log/primary_ca/ca.log,
  
  only the startup messages are seen repeated every few seconds :
  <timestamp> INFO .. [1] OSSM Control Agent starting {"version":
                         "Server Version: 8.1.27.000; Build:
                         20250611_1110; Commit: c55fe83"}
  ...
  <timestamp> INFO .. [1] (23ad6f0ec5a95eda430a) Using Db2
                         database for Storage Protect Plus user
                 {  "db alias": "TSMDB1",
                   "db schema": "OSSM<string>",
                 "db instance": "<instanceName>",
                    "spp user": "23ad6f0ec5a95eda430a",
                   "sp schema": "TSMDB1"}
  ...
  <timestamp> INFO .. [1] db2 schema already exists
                         {"schema name": "OSSM<string>"}
  ...
  
  In the service journal log collected for example here for the
  past 3 days with the command :
  journalctl -u ossm.service --since "3 days ago" > trace.log
  
  The panic message is seen :
  
  <timestamp> <hostname> bash[<pid>]: .. INFO .. [1] OSSM Control
             Agent starting  {"version": "Server Version:
             8.1.27.000; Build: 20250611_1110; Commit: c55fe83"}
  ...
  .. : panic: time: missing Location in call to Date
  .. : goroutine 1 gp=0xc0000061c0 m=12 mp=0xc000900008 [running]:
  
  .. : panic({0xbdf240?, 0xdf5980?})
  .. :      /usr/local/go1.23.2/src/runtime/panic.go:804 +0x168
           fp=0xc000287590 sp=0xc0002874e0 pc=0x4762c8
  .. : time.Date(0xc000522853?, 0xc000522853?, 0xcd9c57?, 0x2?,
           0x7ffff3fc0a68?, 0x18?, 0x2?, 0xc000900008?)
  .. :      /usr/local/go1.23.2/src/time/time.go:1518 +0x4c5
           fp=0xc0002875e8 sp=0xc000287590 pc=0x4a7de5
  .. : time.parse({0xcd9c47, 0x13}, {0xc000522840, 0x1a}, 0x0,0x0)
  
  .. :      /usr/local/go1.23.2/src/time/format.go:1426 +0x2048
           fp=0xc000287898 sp=0xc0002875e8 pc=0x4a2048
  .. : time.ParseInLocation({0xcd9c47, 0x13}, {0xc000522840, 0x1a}
  
           , 0x0)
  .. :      /usr/local/go1.23.2/src/time/format.go:1041 +0xe6
           fp=0xc0002878d8 sp=0xc000287898 pc=0x49ffa6
  .. : github.ibm.com/Spectrum-Protect/sp-ossm-control-agent/pod.
           (*Pod).ExtractSPRestartDate(0xc00047e008?)
  .. :      /extracts/8.1.27.000/srv_git/src/github.ibm.com/
           Spectrum-Protect/sp-ossm-control-agent/pod/heartbeat.
          go:30 +0xf7 fp=0xc0002879c8 sp=0xc0002878d8 pc=0x9a9cf7
  ...
  .. systemd[1]: ossm.service: Main process exited, code=dumped,
                status=6/ABRT
  .. systemd[1]: ossm.service: Failed with result 'core-dump'.
  .. systemd[1]: ossm.service: Scheduled restart job, restart
                counter is at 1.
  .. systemd[1]: Stopped ossm.service.
  .. systemd[1]: Started ossm.service.
  .. bash[<newPID>]: IBM Storage Protect
  .. bash[<newPID>]: Open Snap Store Manager Control Agent
  .. bash[<newPID>]: Server Version: 8.1.27.000; Build:
                    20250611_1110; Commit: c55fe83
  
  This causes the service to be unresponsive to any IBM Storage
  Protect Plus job.
  Example job log messages for a VM backup :
  
  SUMMARY,..,CTGGA2399,Starting job for policy BACKUP with job
                      name vmware_<SLAName>(ID:<SLAID>). id ->
                      <jobID>. IBM Spectrum Protect Plus version
                      10.1.17-152.
  ...  ERROR,..,CTGGA2102,Unable to get storagePools for
                      <OSSMPrimaryAgentHostAddress>. {1}
  ...  ERROR,..,CTGGA2649,Backup of virtual machine (VM) <VMName>
                      has failed.
            Error: Storage volume xxxx.volume.yyy already exists
                   in catalog but unable to get from storage
                   server <OSSMPrimaryAgentHostAddress>.
                   StorageConnectionException
  
  In the corresponding virgo log :
  
  [<timestamp>] INFO ..  Connection error to server I/O error on
                        POST request for "https://
                        <OSSMPrimaryAgentHostAddress>:3337/api/
                        v1beta/auth":
                        Connect to <OSSMPrimaryAgentHostAddress>:
                        3337 [<OSSMPrimaryAgentHostAddress>/<IP>]
                        failed: Connection refused
                        (Connection refused);
                        nested exception is ...
  [<timestamp>] ERROR .. Unable to get storagePools for
                        <OSSMPrimaryAgentHostAddress>. {1}
  
  
  |MDVREGR 8.1.27.000 5698ISMSV|
  
  IBM Spectrum Protect Versions Affected:
  IBM Spectrum Protect Server version 8.1.27.000
  
  Additional Keywords: OSSM, agent, crash, panic, TS020323596
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:                                              *
  * All IBM Storage Protect server users on AIX and Linux        *
  ****************************************************************
  * PROBLEM DESCRIPTION:                                         *
  * See error description.                                       *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  

Problem conclusion

• Changed function from zone to location for time fetching
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  TSM SERVER

• Fixed component ID

  5698ISMSV

Applicable component levels