Data collection modes for the IBM Cloud Pak for AIOps MustGather tool

Learn more about the different commands that you can use with the IBM Cloud Pak for AIOps MustGather tool to gather information for opening a case with IBM Support.

There are 2 sets of data collection modes (primary and secondary). The MustGather tool has multiple (6) primary collection modes and multiple advanced modes for collecting specific sets of data.

Rules:

  • Users can select zero or one primary mode.
  • User can select zero, some or all secondary modes.
  • User can combine one primary mode with zero, some or all secondary modes.
  • Secondary modes can run independently without any primary mode
  • The secondary modes selected (independently, or with any primary mode) are run with the following precedence: clusternodes > cmdexec > manualcollect > extra > missingobj > compliance > plugins > cpfiles > snapshots

Example commands:

waiops-mustgather.sh -aypD -C /tmp/cmdexec.input -m /tmp/manualcollect.csv -e cp4aiops

Since a primary mode -a is selected, all data collection that is configured for the all primary mode is run first. Then the data collection for the secondary modes are run in the sequence of cmdexec > manualcollect > extra.

waiops-mustgather.sh -C /tmp/cmdexec.input -m /tmp/manualcollect.csv -e cp4aiops

Since no primary mode is selected, the secondary modes are run in the sequence cmdexec > manualcollect > extra.

Primary modes

Option Description
-a All mode, which gets all data (data volume is high).
-c Comprehensive mode, which gets comprehensive data (data volume is moderate).
-b Basic mode, which gets basic data (data volume is OK).
-l Limited mode, which gets limited data (data volume is low).
-k Selective mode, which gets data for selective resources (separate resources with comma), for example, pods,deployment,secrets (data volume is varied). To get cluster-wide (all-namespaces) data for namespace-scoped resource, such as a pod, you can use the special keyword allns (pod/allns,ingress,cm/allns). allns data is available in the CLUSTER_DATA directory. namespaced data is available in the PROD_NAMESPACES directory.
-z Module mode, which gets data based on predefined modules separated by comma (clusterinfo productinfo storage networking installation csv pvcusage nodeinfo) (data volume is varied). To list all available modules, use -z .listmodules.

Secondary modes

Clusternode modes

Option Description
-d Gets logs and data from all cluster nodes by using oc debug or SSH (if -i or -u is used).
-l Number of lines of journal to collect from cluster nodes. The default is 2000.

cmdexec mode

Option Description
-C [script file]##[environment file] Where [environment file] is optional.
  • The format of [environment file] is KEY=VALUE, which is similar to the variable definition format in shell script.
-E Environment variable for the cmdexec script. For example, 'VAR1=VAL1##VAR2=VAL2##...'

Enclose the entire input string with single quotation marks. For example 'VAR1=hello##VAR2=uname -a'

-T Timeout value for cmdexec [-C]. The default is 5m. For example, 10s where s=second; m=minute; h=hour.

Manualcollect mode

Option Description
-m <manual-collection-param> = [TAGS]:input file for the getManualCollection() function where TAGS is optional. The default is all config lines.
  • Separate multiple tags with a comma.
-n Environment variables for the manualcollect EXECCMD command. For example, TAG1~>VAR1=VAL1##VAR2=VAL2^^TAG2~>VAR1=VAL1...
  • Use the symbol ^^ to separate environment variables for different tags.
  • Use the symbol ~> to define an environment variable for a tag.
  • Use the symbol ## to separate multiple environment variables.
  • Enclose the entire input string with single quotation marks and each value with double quotation marks. For example, VAR1="hello"##VAR2="uname -a".
-
-j Provided string [tag:<objname>##tag:<objname>##...] is used to substitute the OBJNAME keyword under the OBJNAME column that is based on the tagname in the manual collect input configuration file, such as tag1=pod-abc##tag2=pod-xyz.
-w Timeout value for manualcollect [-m]. The default is 5m. For example, 10s where s=second; m=minute; h=hour.

Extra namespace mode

Option Description
-e Get data for extra namespaces (oc get all,pvc,configmap,secret,serviceaccount, oc describe, oc logs)
  • Separate namespaces with a comma (,)

The [:extra-resouces] is optional and is a file that can be used to specify additional resource types, such as catalog sources, operand configs, and more). Specify one resource type per line in the file.

Missing object mode

Option Description
-x
  • Product version to use for getMissingObj function, such -x 3.2.0 or -x X to auto-detect. Defaults to the product version that is detected by tool if a version is not provided
  • Or, the absolute path for an input config file, such as -x /tmp/myfile.cfg for the getMissingObj function to use.

Compliance mode

Option Description
-A <compliance-param> = TAGS:[input file for the getCompliance() function] where input file is defaulted to compliance/<prod-version>/compliance.cfg (use keyword ALL to select all config lines. Separate multiple tags with a comma) [EXPERIMENTAL]

plugins mode

Option Description
-P Product plugin [supported=aimanager]:action [supported=data fix]. The default action is data. For example, aimanager or aimanager:data or aimanager:fix.
  • Separate plugins with a comma (,).
-Z Environment variables for the plugins script. For example, VAR1=VAL1##VAR2=VAL2##...
  • Enclose the entire input string with single quotation marks. For example, VAR1=hello##VAR2=uname -a.
-Q Timeout value for plugins [-P] The default is 15m. For example, 10s where s=second; m=minute; h=hour.

cpfiles mode

Option Description
-F Copy files in a pod. cpfiles-param = TAGS or NAMESPACE##PODNAME##CONTAINER##FILES where multiple TAGS can be delimited by @@. CONTAINER is optional and FILES can be delimited by a colon.
  • -F aimanager refers to cpfiles/cpfiles-<prodname>-<version>.cfg and copies all the files configured as TAG=aimanager.
  • -F NS4PROD=aimanager##iaf-system-kafka-0####/tmp/abc.txt is the manual command to copy a file.

Notes:

  • You can use keyword NS4PROD=<prodname> to obtain namespace for those products automatically.
  • You can use asterisk(*), square brackets([]), question mark(?), and curly braces({}) wildcards in FILES.
  • Since the tar (needed by oc cp) command might not be available in the target pod or container, use this option only for text files (oc exec/cat is used as a secondary option).

Snapshots mode

Option Description
-K Complete snapshots comparison.

View mode

Option Description
-V <view-only-module> is a script that can be created under the <mustgather-install-dir>/config/view/<PRODUCT_VERSION> (precedence) or <mustgather-install-dir>/view directory. You can use -V .listviews to list down all available modules and their descriptions.

Supporting Options

Option Description
-y Collect YAML output (oc get -o yaml).
-J Collect JSON output (oc get -o json).
-p Collect logs for the previous instance of the container in a pod if it exists
-t Do not collect oc describe.
-g Do not collect pod logs (oc logs and oc logs -p).
-f Collect PVC usage data (df -h)
  • Available only with options -a, -c, -b, -l, -z (productinfo/storage/pvcusage), -k (pvc) and -e.
-R Enable report-on-screen where SUMMARY.log is printed on display.
-S Explicitly collect YAML output of secret objects.
-s Use SSH to collect logs and data from all cluster nodes without a common admin username across the cluster.
-u Provide the admin username to used to SSH to all cluster nodes for data collection purpose. A passwordless SSH is needed for this option.
-i Set the path to the SSH key. This path is needed for clusters on a cloud (experimental).
-o Set the output directory. The default directory is /tmp.
-G Use a filename regex and pattern, that is delimited by a colon, to run a grep operation. This option is applicable to pod logs, events, cpfiles, and node-related data files.
  • Separate patterns with double asperands (@@).
  • Separate FILENAME_REGEX:PATTERNS key value pairs with double hashes (##).
  • Use the keyword ALL as the filename regex as an instruction to run grep for all detected files.
  • Use a tilde (~) to run a piping grep. For example, grep A <file> | grep B | grep C.
  • To escape, use a tilde (~) with a backward slash (\) for the literal value. There is no need to escape by using a colon (:). For example, ALL:error@@failure or ^aimanager:error~fail or ^aimanager:error##ALL:failure or ALL:error\~fail##ALL:error\/fail
-W Set global environment variable that all primary data collection modes can use for further processing.
  • For example, -W 'VAR1=123##VAR2=456##VAR3=789'; this option delimits multiple variables with the ## characters. All variables are suffixed with the string PRIMODE_ when used internally (VAR1 > PRIMODE_VAR1).
-N Turn on data collection for problematic non-product namespaces. Data collection for problematic Red Hat OpenShift namespaces is always enabled.
-I Enable incident reporting (experimental).
-X Turn off custom data collection.
-Y Auto-answer YES to question prompts.
-U Check for updates.
-M Turn of the automatic new version check and download for the tool.
-D Turn on Debug mode.
-v View the version details.
-h View the tool help details.