IBM Support

Collecting Data: IBM Spectrum Protect Server Hangs

Troubleshooting


Problem

Collecting Data documents aid in problem determination and save time resolving Problem Cases (previously called PMRs).

Resolving The Problem

Collecting troubleshooting data early, even before opening the Case, helps IBM® Support quickly determine if:
Symptoms match known problems (rediscovery).
There is a non-defect problem that can be identified and resolved.
There is a defect that identifies a workaround to reduce severity.
Locating root cause can speed development of a code fix.

Collecting data table of contents:

Gathering general information
Manually Gathering General Information
Gathering Spectrum Protect Server Hang Information
Submitting Information to IBM Support
Online Self-Help Resources
Related Information


 
Gathering General Information

Entering general information into an electronically-opened Problem Case (previously called PMRs) eliminates waiting on the phone to provide general information to support. 

 
Manually Gathering General Information

From an IBM Spectrum Protect Admin command line client, enter the following commands:
  • QUERY SYSTEM > querysys.out
  • QUERY ACTLOG begind=<mm/dd/yyyy> begint=<hh:mm> endd=<mm/dd/yyyy> endt=<hh:mm> > actlog.out

  • - where begind and begint are the beginning date and time for the actlog entries being collected
    - where endd and endt are the ending date and time for the actlog entries being collected
    - the actlog gather should cover the full time frame of the issue/problem/scenario being diagnosed

    Explicitly using the above commands will redirect the output to a file called querysys.out and actlog.out in the Spectrum Protect server's working directory. The names of these files can be changed and a full path can be specified to place the output in any desired directory using any desired name.

These files along with the following files/info should be included as general information:
  • dsmserv.opt
  • dsmserv.err
  • details of operating system levels
  • IBM Spectrum Protect Server specific version (ex: 8.1.0.2)

On IBM Spectrum Protect servers please also gather/upload the:
1) DB2diag.log
2) run : db2fodc -hang
3) Database Reorganizations may also contribute to server hangs. You may also gather reorganization information from the following document:
Collecting Data for IBM Spectrum Server Database Reorganization
http://www-01.ibm.com/support/docview.wss?uid=swg21590928
 
Gathering Spectrum Protect Server Hang Information
[Questions to answer:1) Is there high CPU usage, or is CPU higher than what is expected or typically experienced? Yes/No

If yes, there could be a loop.


If no, continue
2) Can new sessions start:
A) Administrative/dsmadmc Yes/No
B) Client Yes/No
3) Can previously established sessions still run commands: Yes/no
If "yes" to question 2 or 3 go to the section – "Show Commands (Undocumented)"
Else go to the Dump/Debugger section
4) Does the hang clear itself?
How long did the hang last?
Does this coincide with the resourcetimeout server setting?

IBM Spectrum Protect Show Commands (Undocumented)
- Show commands are very helpful in further problem determination. Gathering this documentation can speed problem resolution.

- The following "Show" commands should be redirected to a file and be gathered in the following order:
Show Resq (Optional) usually recommended when gathering show commands at specified intervals such as every 10 minutes.
Show Deadlock
Show Locks
Show TXNT
Show Threads
Show Session

Dump/Debugger
- When the IBM Spectrum Protect server is no longer responsive, it becomes necessary to force a core dump and attempt to read the thread output using a debugger such as DBX, GDB, ADB, etc..
- To force a core dump on a Unix/Linux system issue a: Kill -11 PID (Process ID for dsmserv).
- By default a file called “core” will be created in the IBM Spectrum Protect server install directory.
- Good practice is to rename or move this file to keep it from being overwritten.
- Go to the next section "Using a Debugger".
- To force an IBM Spectrum Protect core dump on a Windows system issue either:
1) adsmtrap.exe This only works if the IBM Spectrum Protect Server is started in the foreground, not as a service. Note: "c:\program files\tivoli\tsm\server" is not in the Path, so CD to this directory first.
2) A second/simpler method with Windows, is to start Task Manager, go on the Process tab, select "dsmsvc.exe" (or "dsmserv.exe" for foreground), right-click on it and select "Create dump file". The location of the dump file will be provided on a Pop-up message once it has been written to disk.
- Send this dump file (see the section on Submitting information to IBM Support)

Using a Debugger
- All debuggers have similar capabilities but use different commands to accomplish the task at hand. For purposes of this technote the DBX debugger will be used.
- To locate a debugger issue the "which" command: which DBX (as an example)
- Note: For AIX, should the DBX debugger (bos.adt.debug) needs to be installed. It is usually on AIX install CD #1 - base and does not require reboot to install
1) Run DBX from the IBM Spectrum Protect server installation directory:
dbx dsmserv /path/core-file-name
dbx prompt Type: where
If a message similar to "cant read address" is displayed then the core is truncated and may not be usable.
Else DBX will produce a call stack (save this)
Issue from DBX: thread (save and send this output )
To quit DBX type: quit
2) If not truncated TAR/Zip core, actlog, dsmserv.err, OS error report, and the outputs from dbx commands.
3) Upload the files to your case.

If a debugger is not available and can not be installed then the dependent files and libraries will need to be gathered. See the Problem Determination Guide > Resolving a Server Crash for a list of files and instructions:

AIX has a utility called snapcore that will gather the dependent files and libraries with the core. See Knowledgebase document 1319850 for instructions.


 
Submitting Information to IBM Support

After a Case is open, you can submit diagnostic data to IBM.

When data is uploaded to the Case, the status will be automatically be updated to acknowledge that data has been received. If you are still working with older Problem Management Records, you will want to update the PMR to indicate that you have sent the data.
 
Online Self-Help Resources
 
Related Information

[{"Product":{"code":"SSEQVQ","label":"IBM Spectrum Protect"},"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Supported Versions","Edition":"Edition Independent","Line of Business":{"code":"LOB26","label":"Storage"}}]

Product Synonym

TSM

Document Information

Modified date:
28 October 2019

UID

swg21264480