SevOne NMS MustGather Evidence Collection Guide
Summary
IBM SevOne NMS issues that require the technical support engagement on an IBM Support Case, requires the collection of evidence data from the appliances of the NMS cluster. The Command Line Interface tool provides a standard method to collect the MustGather data and important evidence required on a support case.
Objective
This article explains how to run the SevOne NMS support tools for evidence collection on the NMS appliances.
Environment
The tool can run on any of the following SevOne NMS appliances including the SDN,
SD-WAN, and Wi-Fi plugins.
- PAS - Performance Appliance System
- DNC - Dedicated NetFlow Collector
- HSA - Hot Standby Appliances
MustGather Data Collection
Follow the steps in connect to the Command Line Interface on IBM SevOne NMS appliance to collect the data.
SYNOPSIS
SevOne-act support command [options]
DESCRIPTION
SevOne-act support is a command-line toolkit for SevOne NPM (NMS) appliances that provides commands to perform different support-related tasks. For example, collection of evidence that uses the collect command.
COMMAND
collect - The collect command performs the collection of evidence for the troubleshooting and investigation of the NMS. The collection is in the form of necessary logs, configuration files, command outputs, and database information to facilitate the troubleshooting of a problem in a consistent way.
GENERAL OPTIONS
-v, --verbose (Optional) Enable verbose logging. Shows all of the debug logging output of the operation, useful for debugging/troubleshooting
-h, --help (Optional) Print this help message, more info with --verbose
COMMAND 'collect' SPECIFIC OPTIONS
-d, --modules (Optional) alerts, all, api, clustering, device-discovery, device-polling, must-gather, mysql,
network-flow, reporting, sdp, solutions,sdn, wifi, sdwan, ssu, traps.
-o, --output-dir (Optional) This is used to override the default output directory location where the outputs will be
collected.
Default: /var/support
-c, --include-compressed(Optional) By default, compressed or logrotated files are excluded from collection. Set this option to
'true' to include collection of the compressed/logrotated files. Example: '--include-compressed true ||
-c true'
Default: false
-v, --verbose (Optional) This option is to provide a verbose output of the operation. It will list the detailed steps performed.
LIST OF MODULES
| MODULE | DESCRIPTION |
|---|---|
| alerts | This module collects information from the NMS relevant to the alerts in the NMS. |
| all | This module collects information for all available modules as a single operation. |
| api | This module collects information from the NMS relevant to the SevOne API (REST API v3). |
| clustering | This module collects information specific to the clustering operations (peer add/remove, HSA add/remove) in the NMS. |
| device-discovery | This module collects information specific to the device discovery (non-Flow) operations in the NMS. |
| device-polling | This module collects information specific to the device polling (non-Flow and excluding external plugins like xStats and Deferred Data) operations in the NMS. |
| must-gather (default) | The must-gather is the default module for the evidence collection command, collect. This option is always selected and included during evidence collection. This module collects the general system, platform, and system resource information about the NMS appliance. |
| mysql | This module collects information specific to the MySQL database (including replication) operations in the NMS. |
| network-flow | This module collects information specific to the network Flow operations in the NMS. |
| reporting | This module collects information specific to the reporting operations in the NMS. |
| sdp | This module collects information for SevOne Data Publisher in the NMS. |
| sdn | This module collects information required for the SDN Solution for operations in the NMS. |
| sdwan | This module collects information required for the SD-WAN plugin-based Solution for operations
in the NMS.
Note: This collection is only specific to the solutions related information on the NMS
appliance including the SD-WAN plugins. To perform collection on the SD-WAN collectors -
Versa/Viptela, please refer to the respective product documentation.
|
| solutions | This module collects generic information specific to the SevOne NPM Solutions (SD-WAN,
SDN and Wi-Fi) operations in the NMS. Note: This collection is only specific to the
solutions related information on the NMS appliance. To perform collection on the Solutions
collectors, please refer to the specific modules.
|
| ssu | This module collects information specific to the Self Service Upgrade (SSU) operations in the NMS. |
| traps | This module collects information from the NMS relevant to the traps in the NMS. |
| wifi | This module collects information required for the Wi-Fi Solution for operations in the NMS. |
EXAMPLES
## This command will provide the general information about the Support Toolkit and the available commands
SevOne-act support
## This command will perform the collection of evidence (Log or configuration files/directories, command outputs and Database information).
## Since no options are selected, it will use the default values for the options.
SevOne-act support collect
## This command will perform the collection of evidence for the specified module 'must-gather'.
SevOne-act support collect --modules must-gather
## This command will perform the collection of evidence for the specified modules for 'clustering' and 'mysql'.
## Note: The 'must-gather' will always be collected even if it is not specified.
SevOne-act support collect --modules clustering,mysql
## This command will perform the collection of evidence for all available modules.
SevOne-act support collect --modules all -v
## This command will perform the collection of evidence to an alternate location than the default.
The output collection directory will be /tmp in this case.
SevOne-act support collect -o /tmp
OR
SevOne-act support collect --modules must-gather -o /tmp
## This command will perform the collection of evidence for the api and mysql modules and include all compressed log files.
SevOne-act support collect --modules api,mysql --include-compressed true
OR
SevOne-act support collect --modules api,mysql -c true
## This command will perform the collection of evidence for all available modules and log the detailed verbose output of all the steps performed.
SevOne-act support collect --modules all -v
## This command will perform the collection of evidence for all available modules and include all compressed versions of the logs.
SevOne-act support collect --modules all -c true
OR
SevOne-act support collect --modules all --include-compressed true
Execute the collect command to gather the data. There are three scenarios explained in this article. Choose the scenario based on the requirement to collect data either on a single appliance, on a few appliances, or on all appliances cluster wide. Follow the steps outlined specific to your requirement and specifying the required modules and the collection directory location if necessary.
Example Execution
root@ibm-nms-01:~ [8.0.0] [12:38:22] $ SevOne-act support collect --modules must-gather
Compressed files will be excluded during the collection.
Directory already exists: /var/support.
Collection Directory: /var/support/ibm-nms-01
The following modules will be collected: must-gather.
Start Processing 'path' task...
Task 'path' file: /var/support/ibm-nms-01/path.csv
Completed Processing task 'path' ...
Start Processing 'command' task...
Task 'command' file: /var/support/ibm-nms-01/command.csv
Completed Processing task 'command' ...
Start Processing 'dbquery' task...
Task 'dbquery' file: /var/support/ibm-nms-01/dbquery.csv
Task 'dbquery' file:
Completed Processing task 'dbquery' ...
Executing Command: 'cd /var/support; tar -zcvf ibm-nms-01-support-collect-20250718-153925.tar.gz ./ibm-nms-01'
==================================================================================================
Collection Complete
==================================================================================================
The compressed file is: /var/support/ibm-nms-01-support-collect-20250718-153925.tar.gz
Please kindly upload the resulting output file to the support case.
'/var/support/ibm-nms-01-support-collect-20250718-153925.tar.gz' Size: 11203773 bytes / 10.68 MB
'/var/support/ibm-nms-01-support-collect-20250718-153925.tar.gz' sha1: 3e56d6e268675a6954e90f6929e0309201b03596
==================================================================================================
How to upload files to cases: https://www.ibm.com/mysupport/s/article/How-to-Upload-Files-to-Cases
==================================================================================================
Removing working directory path: '/var/support/ibm-nms-01'.
Directory removed successfully: '/var/support/ibm-nms-01'.
Logging output to toLogs
[ OK ] No Errors Detected
SCENARIO# 1 - Steps to perform the evidence collection on a single appliance.
- Set a variable with a log file name to log the execution output (including errors, if any).
root@ibm-nms-01:~ [8.0.0] [12:38:22] $ LOGFILE="/var/support/support-collect-$(hostname)-$(date +%y%m%d%H%M%S)" root@ibm-nms-01:~ [8.0.0] [12:38:28] $ echo $LOGFILE /var/support/support-collect-ibm-nms-01-230718123828 - Execute the command on the appliance where you want to collect the evidence data.
Choose the different modules and options as necessary at the time of execution.
root@ibm-nms-01:~ [8.0.0] [12:39:53] $ SevOne-act support collect 2>&1 | tee -a $LOGFILE Directory already exists: /var/support. Collection Directory: /var/support/ibm-nms-01 The following modules will be collected: must-gather. Start Processing 'path' task... Task 'path' file: /var/support/ibm-nms-01/path.csv Warning: This is an invalid path: '/var/log/oom-kill.log'. Not a file or directory. Completed Processing task 'path' ... Start Processing 'command' task... Task 'command' file: /var/support/ibm-nms-01/command.csv Completed Processing task 'command' ... Start Processing 'dbquery' task... Task 'dbquery' file: /var/support/ibm-nms-01/dbquery.csv Completed Processing task 'dbquery' ... Executing Command: 'cd /var/support; tar -zcvf ibm-nms-01-support-collect-20230714-124008.tar.gz ./ibm-nms-01' ================================================================================================== Collection Complete ================================================================================================== Please kindly upload the resulting output file to the support case. '/var/support/ibm-nms-01-support-collect-20230714-124008.tar.gz' Size: 11061853 bytes / 10.55 MB '/var/support/ibm-nms-01-support-collect-20230714-124008.tar.gz' sha1: b9cb55b2faf7d2679dd46a0d00e79f3a95012afe How to upload files to cases: https://www.ibm.com/mysupport/s/article/How-to-Upload-Files-to-Cases ================================================================================================== Removing working directory path: '/var/support/ibm-nms-01'. Directory removed successfully: '/var/support/ibm-nms-01'. Logging output to toLogs [ OK ] No Errors Detected - Collect and upload the log file and the collected evidence file to the Support Case. The Log File name is in bullet 1. The file name and details of the support evidence file is displayed on the output from the collection. Upload both these files to the support case.
- You can remove the files after they are received, verified, and reviewed by the Support
engineer. Change the directory location if a different path is selected at the time of execution.
root@ibm-nms-01:~ [8.0.0] [14:51:18] $ rm /var/support/*support-collect*.tar.gz
SCENARIO# 2 - Steps to perform the evidence collection on more than one appliance.
- On the Cluster Leader appliance, set a variable with the list of comma-separated IPv4
addresses of the NMS appliances for evidence data collection. NOTE: Replace your appliance's
IPs in the command.
root@ibm-nms-01:~ [8.0.0] [12:44:23] $ IPS='192.168.1.1,192.168.1.2,192.168.1.3' - Set a variable with a log file name to log the execution output (including errors, if any).
Take a note of this log file to send along with the collected output file.
root@ibm-nms-01:~ [8.0.0] [12:44:05] $ LOGFILE="/var/support/support-collect-$(hostname)-$(date +%y%m%d%H%M%S)" root@ibm-nms-01:~ [8.0.0] [12:44:12] $ echo $LOGFILE /var/support/support-collect-ibm-nms-01-230718124412 - Execute the command on the Cluster Leader appliance. Choose the different modules and options as
necessary at the time of execution. The execution output is similar to the execution on the single
appliance, one output per appliance. So this output can be large if executed on many appliances.
root@ibm-nms-01:~ [8.0.0] [14:46:46] $ ansible all -i "mkdir -p /tmp/.ansible/cp ; $IPS," -T 60 -f 200 -m shell -a "SevOne-act support collect" 2>&1 | tee -a $LOGFILE - Gather the output file from each appliance and collect them on the cluster master. If a different
directory is selected at run time, then replace the directory from /var/support to the output
directory location in the command, and in the next steps.
root@ibm-nms-01:~ [8.0.0] [14:47:05] $ ansible all -i "$IPS," -T 60 -f 200 -m shell -a "scp -r /var/support/*support-collect*.tar.gz $(SevOne-peer-list --master | xargs):/var/support" - Collect and upload log file and the collected evidence files from all the selected appliances
to the Support Case. The output files for all the appliances are stored in /var/support by
default, or in the chosen output directory. You can upload the individual files to the support
case, or you can tar all the appliance's files gathered in step 4, and then upload only one file
to the case instead of multiple files. To tar all files in one, execute the following command.
Upload the resulting tar file /var/support/cluster-support-collect.tar or your chosen path to the support case. Upload the Log File from the path as defined in step 2.root@ibm-nms-01:~ [8.0.0] [14:50:09] $ tar --remove-files -cvf /var/support/cluster-support-collect.tar /var/support/*support-collect*.tar.gz - You can remove the files after they are received, verified, and reviewed by the Support
engineer. Change the directory location if a different path is selected at the time of execution.
root@ibm-nms-01:~ [8.0.0] [14:51:18] $ ansible all -i "$IPS," -T 60 -f 200 -m shell -a "rm /var/support/*support-collect*.tar.gz"
SCENARIO# 3 - Steps to perform the evidence collection on all appliances, cluster-wide.
- Set a variable with a log file name to log the execution output (including errors, if any).
root@ibm-nms-01:~ [8.0.0] [14:53:37] $ LOGFILE="/var/support/support-collect-$(hostname)-$(date +%y%m%d%H%M%S)" root@ibm-nms-01:~ [8.0.0] [12:53:48] $ echo $LOGFILE /var/support/support-collect-ibm-nms-01-230718125348 - Execute the command on the Cluster Leader appliance. Choose the different options as necessary
at the time of execution. The execution output is similar to the execution on the single appliance,
one output per appliance. So this output can be large if executed on many appliances.
root@ibm-nms-01:~ [8.0.0] [14:55:06] $ SevOne-peer-do --asynchronous "SevOne-act support collect" 2>&1 | tee -a $LOGFILE - Gather the log file from each appliance and collect them on the cluster master. If a
different directory is selected at run time, then replace the directory from /var/support
to the output directory location in the command, and in the next steps.
root@ibm-nms-01:~ [8.0.0] [14:56:19] $ SevOne-peer-do --asynchronous "scp -r /var/support/*support-collect*.tar.gz $(SevOne-peer-list --master | xargs):/var/support" - Collect and upload log file and the collected evidence files from all the selected appliances
to the Support Case. The output files for all the appliances are stored in /var/support
by default, or in the chosen output directory. You can upload the individual files to the
support case, or you can tar all the appliance's files gathered in step 3, and then upload
only one file to the case instead of multiple files. To tar all files in one, execute the
following command.
Upload the resulting tar file /var/support/cluster-support-collect.tar (or your chosen path) to the support case. Upload the Log File from the path as defined in step 3.root@ibm-nms-01:~ [8.0.0] [14:57:11] $ tar --remove-files -cvf /var/support/cluster-support-collect.tar /var/support/*support-collect*.tar.gz - You can remove the files after they are received, verified, and reviewed by the Support
engineer. Change the directory location if a different path is selected at the time of execution.
root@ibm-nms-01:~ [8.0.0] [14:59:24] $ SevOne-peer-do --asynchronous "rm /var/support/*support-collect*.tar.gz"
Important:
Ensure that you upload both, the support evidence file and the log file to the support case.
References
| NAME | URL |
|---|---|
| How to Upload Files to Support Cases? | https://www.ibm.com/mysupport/s/article/How-to-Upload-Files-to-Cases |
| Enhanced Customer Data Repository (Ecu Rep) - Send data (HTTPS) | https://www.ibm.com/support/pages/enhanced-customer-data-repository-ecurep-send-data-https |
| How to transfer files from the SevOne NMS appliance? | https://www.ibm.com/support/pages/node/7159843 |