About cookies on this site Our websites require some cookies to function properly (required). In addition, other cookies may be used with your consent to analyze site usage, improve the user experience and for advertising. For more information, please review your options. By visiting our website, you agree to our processing of information as described in IBM’sprivacy statement. To provide a smooth navigation, your cookie preferences will be shared across the IBM web domains listed here.
Troubleshooting
Problem
How do I collect troubleshooting data for the IBM Spectrum Protect server when encountering problems with server processes, client sessions and/or of a performance nature?
Environment
Please use version 3 here: http://www-01.ibm.com/support/docview.wss?uid=swg21432937
Version 2 found in this technote is obsolete, only use it if directed by support personnel.
For Linux x86 servers running Spectrum Protect 8.1.6.0 only:
Please disable instrumentation editing servermon.ini and changing the value of "instrumentation" to N :
Example:
instrumentation = N
Other platforms and versions do not have an issue with instrumentation and should remain enabled.
Resolving The Problem
Collecting Data: Table of contents:
servermon.pl is a Perl script used to collect data from IBM Spectrum Protect servers. It typically does so in 20 minute intervals called diagCycles. But there are also additional collections as when the script starts and ends.
The script can also be used to collect a trace and in addition to that, stop the collection when a specific server message is logged in the activity log.
The script primarily collects IBM Spectrum Protect specific data such as server instrumentation for thread level analysis, various queries and show commands used for troubleshooting. It also collects DB2 specific data for when it's necessary to do more in depth analysis on a server problem. And finally, it also collects data from the operating system, such as CPU, memory and disk metrics.
This script has a very low overhead. It has been tested both internally and with customers and is not known to cause performance degradation.
With this technote you can find the most current version of the server monitoring script. This is supported at all levels of IBM Spectrum Protect server V7.1 and higher. To set up monitoring you need the following 3 files from the downloads section:
- servermon.pl - the script file itself
- servermon.ini - configuration parameters
- commands.ini - commands that are submitted by the script
Download the files to a local directory, Data collected will be written to the current directory from where the script is run.
The script can be run a few different ways:
- in the foreground (default)
- in the background (noprompt parameter)
- on a storage agent (no database)
Note: The script requires a Perl interpreter in order to run. Some operations systems such as Windows do not ship with Perl installed. Strawberry Perl and Active Perl are two popular distributions, but there are others as well.
1. Download the 3 files (servermon.pl, commands.ini and servermon.ini) to a local directory, data collected will be written to the current directory from where the script is run.
2. For the script to collect a complete set of docs on an IBM Spectrum Protect server, it must be run locally as the instance user.
Windows:
Run the script from a DB2 command window:
Start -> Run -> db2cmd
From there, cd to the target directory where servermon.pl was saved and issue this command:
perl servermon.pl
AIX:
Change directory to where servermon.pl is stored and issue:
perl64 servermon.pl
(the perl64 interpreter prevents an out of memory symptom on AIX systems)
Linux:
Change directory to where servermon.pl is stored and issue:
perl servermon.pl
or
./servermon.pl
3. Follow the prompts which are listed below with their explanation:
Enter the servername as found in dsm.opt or dsm.sys [localhost]:
On Windows server specify the TCP/IP address of the target server, as the script is expected to be run on the same machine as the server is running usually localhost will work.
For all non-Windows environments, specify the servername as found in dsm.sys/dsm.opt.
Enter the tcpport [1500]:
On Windows servers you will be prompted for the port the server is listening for dsmadmc commands.
For all non-Windows servers this parameter is not required as the definitions are taken from the server stanza.
Enter the IBM Spectrum Protect administrator login ID [admin]:
Specify the IBM Spectrum Protect administrator id to collect the command output with. This must be an administrator that has been defined with system privilege class authority.
Enter the IBM Spectrum Protect administrator password (no echo to the screen) [password]:
Specify the current password for the IBM Spectrum Protect administrator.
Enter PMR number as PPPPP.BBB.CCC [none]:
If the doc collection is for a PMR you can specify PMR number, branch and country and the collected docs will be put together for easier upload to the PMR doc repository. Enter the PMR number with periods (".") between the PMR number, branch, and country code.
Collect SHOW THREADS output? (Y/N) [Y]:
Specifies whether or not to run the "show threads" command. Default value is Y.
Collect SHOW DEDUPDELETE output? (Y/N) [Y]:
If running legacy deduplication (DEVCLASS=FILE sequential pools), specify Y for this option. If you are only running container pools enter N.
Collect SHOW Collect SHOW for memory statistics? (Y/N) [N]:
SHOW ALLOC/MEM/MEMTREND output is not collected by default, but the information is helpful if the symptom to investigate is memory usage related. Please use as advised by IBM support.
Enable collectstmt instrumentation? (N/Y) [Y]:
Enabling COLLECTSTMT will result in the generation of instrumentation statistics for long running SQL statements. Default value is Y.
Enable SNAPSHOT FOR DYNAMIC SQL? (N/Y) [N]:
Enabling SNAPSHOT FOR DYNAMIC SQL will result in the generation of additional DB2 statistics. As the collection could impact server performance the default is set to not collect the statistics.
Note: usually collecting snapshot for dynamic SQL requires to enable certain DB2 monitors, please use as advised by IBM support.
Monitor log spanning? (N/Y) [Y]:
Log span monitoring is enabled by default, the information gathered can help to analyze transactions that use lots of active log space.
Enter comma separated stop events, e.g. ANR#####E [none]:
Here you can specify a comma separated list of messages as logged to the activity log. If any of the messages specified is found during the runtime of the script, the script will go through some final doc collection and then stop.
Enter whitespace separated traceflags [none]:
Here you can specify any server traceflags as asked for by IBM support. Tracing will be stopped upon stopping the script via sending SIGINT or if the number of diagnostic cycles are reached (see below).
Enter traceflags to disable [none]:
Here you can specify any server traceflags to disable, for e.g. if you specified trace aggregate AF you might want to disable AFTXN subset. Specify as instructed by IBM support.
Enter size limit for the tracefile. (1-65535MB, -1 unlimited) [-1]:
If you enabled tracing via the traceflags parameter you can specify the maxsize limit in MB for the generated tracefile.
Copy FFDCLOG file? (N/Y) [Y]:
At the end of each (documentation collection cycle (see below) add a copy of the FFDCLOG file to the collected docs. Default is "Y". This is only supported for IBM Spectrum Protect servers, not storage agents.
Enter number for diagnostic cycles of 1200s to run? (72 = 1 day, indefinite if -1) [72]:
Here you can specify how many diagnostic cycles you want to collect documentation for, e.g. 3 cycles would collect data for ~ 60 minutes. As gathering the docs can take up some time, the time until script completion might differ.
Only if you specify a positive value for diagnostic cycles, you will get prompted for collection cycles.
The following prompt example is for 72 diagnostic cycles:
Enter number of 86400s documentation collection cycles to run? (1 = 1 day, indefinite if -1) [-1]:
This option can be used to configure some long term monitoring. In the above example (72 diagnostic cycles), if for example you specify 7 collection cycles, monitoring will be active for about 7 days. Again, as gathering the docs can take up some time, the time until script completion might differ. Once the number of diagnostic cycles (see above) is reached, the docs collected during this collection cycle will get zipped and a new collection will get started. Basically with the number of collection cycles you specify a loop around diagnostic cycles.
1. Download the 3 files (servermon.pl, commands.ini and servermon.ini) to a local directory, data collected will be written to the current directory from where the script is run.
2. To run in the background, you need to update some of options in the servermon.ini file. The servermon.ini file contains different configuration stanzas, At a minimum the following options in the [TSM] stanza need to be updated:
[TSM]
server = SERVERSTANZA # server stanza for UNIX,TCPSERVERADDRESS for Windows
tcpport = 1500 # only used on Windows
administrator = ADMIN # the admin id
password = PASSWORD # guess what?
The servermon.ini file assigns values to a fixed set of variables and is of the form:
variable = value # some comment
To find the proper value to use for the server variable, for *IX systems look at the server stanza that is either configured in the default dsm.sys file or the file DSM_CONFIG environment variable is pointing to. On Windows systems, you can specify the IP address of the server or alternatively specify either localhost or 127.0.0.1. The value for tcpport is the port number the server listens to.
3. For the script to collect a complete set of docs on an IBM Spectrum Protect server, it must be run locally as the instance user.
Windows:
Run the script from a DB2 command window:
Start -> Run -> db2cmd
From there, cd to the target directory where servermon.pl was saved and issue this command:
perl servermon.pl noprompt
AIX:
Change directory to where servermon.pl is stored and issue:
perl64 servermon.pl noprompt
(the perl64 interpreter prevents from an out of memory symptom on AIX systems)
Linux:
Change directory to where servermon.pl is stored and issue:
perl servermon.pl noprompt
or
./servermon.pl noprompt
1. To collect monitoring data for an IBM Spectrum Protect storage agent (STA), review the client dsm.sys or dsm.opt file for the following options:
- Lanfreecommmethod (supported on *IX via server stanza)
- Lanfreeshmport (supported in *IX via server stanza)
- Lanfreetcpport
- Lanfreetcpserveraddress
2. On *IX systems, create a server stanza in dsm.sys for storage agent that matches the communication definitions identified in Step 1.
TCPServeraddress -> Lanfreetcpserveraddress
TCPPort -> Lanfreetcpport
Commmethod -> Lanfreecommmethod
Shmport -> Lanfreeshmport
For Windows systems, you need to configure the STA to allow for TCP communication by adding "commmethod tcpip" in dsmsta.opt if it it's not present in order for the script to work properly.
3. Download the 3 files (servermon.pl, commands.ini and servermon.ini) to a local directory, data collected will be written to the current directory from where the script is run.
4. To start the script in the foreground no additional parameter is needed, to start the script in the background, do the following:
Windows:
From a Windows Command Prompt, change directory to where servermon.pl is stored and issue: and issue this command:
perl servermon.pl
AIX:
Change directory to where servermon.pl is stored and issue:
perl64 servermon.pl
(the perl64 interpreter prevents from an out of memory symptom on AIX systems)
Linux:
Change directory to where servermon.pl is stored and issue:
perl servermon.pl
or
./servermon.pl
5. Follow the prompts
Enter the servername as found in dsm.opt or dsm.sys [localhost]:
On Windows server specify the TCP/IP address of the storage agent, as the script is expected to be run on the same machine as the server is running usually localhost will work.
For all non-Windows environments, specify the server stanza for the storage agent as found in dsm.sys/dsm.opt.
Enter the tcpport [1500]:
On Windows servers you will be prompted for the port the storage agent is listening for dsmadmc commands.
For all non-Windows servers this parameter is not required as the definitions are taken from the server stanza.
Refer to " running servermon in foreground " section for complete list of prompts.
1. If the script is run with noprompt with a set number of diagnostic/collection cycles, it could also be used with a task scheduler (making sure that the DB2 environment is being established correctly first). This way, the script could be run continuously on a day to day basis to gather performance data or monitoring information for a given problem. In addition, the corresponding files created can be also be archived off (and deleted) with IBM Spectrum Protect for historical purposes using the archive retention policies set up in archive copygroup.
2. You must change the [DB2] stanza if your instance name is something other than the default of tsminst1:
[DB2]
instance = TSMINST1 # DB2 instance name, if your DB2INSTANCE environment variable is set it
dbalias = tsmdb1 # DB2 database alias for TSM database
When the script is invoked it will verify that the instance specified exists, and if not the script will provide a list of instances to select from.
In most cases, the script will create a .zip file in the current directory, that is the file that needs to be sent to support.
In some cases if the script is interrupted before it completes the collection cycle, then it will be necessary to zip or tar.Z the data collection directory located in the current directory. The directory has the following naming convention: YYYYMMDD-HHMM-swg21590928
Data collected can be submitted to IBM using either ECUREP or Blue Diamond depending on the account..
If using ESR , update the PMR to indicate that data has been sent.
Review up-to-date product information at the IBM Spectrum Protrect Product Support page .Utilize the IBM Electronic Service Request tool to access the Spectrum Protect Support team when requiring assistance from IBM.For the 8.1.6 server servermon technical preview see Troubleshoot issues in an IBM Spectrum Protect environment by using the servermon commponent .
IBM Spectrum Protect Knowledge CenterIBM Spectrum Protect Performance Best PracticesIBM Spectrum Protect BlueprintIBM Spectrum Protect Container Pools Best Practices
See the attachment section below
Related Information
[{"Business Unit":{"code":"BU058","label":"IBM Infrastructure w\/TPS"},"Product":{"code":"SSGSG7","label":"Tivoli Storage Manager"},"Component":"Server","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Supported Versions","Edition":"","Line of Business":{"code":"LOB26","label":"Storage"}}]
Was this topic helpful?
Document Information
More support for:
Tivoli Storage Manager
Software version:
All Supported Versions
Document number:
795810
Modified date:
18 January 2019
UID
ibm10795810