Collecting support information about the cluster

The MustGather tool collects information about your cluster that is crucial for troubleshooting problems and providing support.

The MustGather tool collects support information about the IBM Cloud Paks and the clusters where they are deployed. You can use this tool to gather information that is needed and relevant to the problem before you open a case with the IBM® Support Opens in a new tab.

The MustGather tool is built on the OpenShift MustGather tool Opens in a new tab.

Using MustGather diagnostics through the command line for cluster admin

Important: IBM Cloud Pak foundational services version 3.18 onwards, the MustGather image for IBM Cloud Paks is available at a new location. If you use the latest tag for the MustGather image, the image from the new location is supported on all versions of foundational services.

If you need to use a specific image version, see the following notes:

All references and commands in the following sections point to the icr.io/cpopen/cpfs/ location. If your foundational services version is 3.17 or prior, and you need to use a specific image version, you can replace icr.io/cpopen/cpfs/ with quay.io/opencloudio/, where required.

Available data modules

IMPORTANT: Non-cluster admin supports Overview, Failure and Cloudpak data modules. Cluster admin supports all the available data modules.

To collect a specific type of support information only, define the data modules when you run the MustGather tool. The following table lists the details of all available modules.

Note: The default scan does not collect information about secrets as it is a user sensitive data. To collect information about secrets, you can use the dedicated data module.

Table 1. Available data modules
Data module Collected cluster data Included in the default scan?
overview
  • Version
  • List of nodes
  • List of all basic Kubernetes resource
  • List of secrets without secret details
  • Cluster version update status and related logs
Yes
system
  • /proc/cpuinfo on master nodes
  • /proc/meminfo on master nodes
  • dmesg on master nodes
  • IP address on master nodes
  • iptables, ovs-vswitchd data and log on master nodes
  • df -h on master nodes
  • kubelet and CRI-O log on master nodes
Note: If you gather the MustGather diagnostics from Administration panel or OpenShift cluster console, you cannot collect the cluster details that are defined by the system data module because of the restricted Security Context Constraints.
Yes
failure
  • Unhealthy pods and their logs
  • Unhealthy deployments, daemonsets, statefulsets, and jobs
  • Unhealthy routes and services
  • Unhealthy pv and pvc
  • Unhealthy namespaces
Yes
ocp
  • The OpenShift MustGather data, the yaml and logs related to OpenShift (no audit logs)
  • kubelet and CRI-O log on master pods
Yes
cloudpak
  • Cluster scope resources: nodes, clusterroles, clusterrolebindings, storageclasses, persistentvolumes, volumeattachments
  • Namespace scope resources: user-defined namespaces, data that is collected by oc adm inspect ns/<namespace>
  • Label scope resources: resource data that matches the user-defined labels
Yes
secrets
  • oc describe secrets in user-defined namespaces
No
etcd
Available for OpenShift version 4.5 and higher
  • etcd member list
  • etcd endpoint health
  • etcd endpoint status
No
route
  • Route pod resource yaml and log
  • haproxy.config of route pods
No

Prerequisites

Quick start

Complete the following steps to retrieve the support information from your cluster.

  1. Navigate to the directory where you want to store the MustGather data.
  2. Collect the data by running the following command. The <image_version> is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

     oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n common-service,<your-foundational-services-namespace>
    

    For example:

     oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n common-service,<your-foundational-services-namespace>
    

The collected data is compressed into a .tgz output file and stored in ./must-gather.local.<rand>.

  1. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal Opens in a new tab.

Usage and examples- cluster admin

Available command options

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

oc adm must-gather --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
oc adm must-gather command options
Table 2. oc adm must-gather command options
Option Description
--image A MustGather plugin image to run. Specify the image name and version, for example, icr.io/cpopen/cpfs/must-gather:4.5.16.

Note: oc adm must-gather is a default OpenShift command. Run the oc adm must-gather -h command for more information.

gather command options:

Option|Description |---|---| | -m | Define the data module that specifies what type of information is collected. For more information, see Available data modules. When this option is not set, the MustGather tool collects information that includes the following data modules: overview, system, failure, ocp, cloudpak. | -n
--namespaces= | Specify the IBM Cloud Pak namespace or namespaces from which the data is collected for the cloudpacks and secrets data modules. By default, the namespace is set to foundational-services. Separate each namespace with a comma, for example <your-foundational-services-namespace>,common-service. | | -h | Display the help message.| | -v | Display the version.| | -l | Specify the IBM Cloud Pak labels to collect logs. Separate the labels with a comma. This option requires you to set the -m cloudpak at the same time. By default, no labels are defined.| | -c
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Specify the component name to collect logs. If you pass the component-name followed by -c parameter, must-gather collects the MustGather diagnostics with the label serviceability-addon:<component-name>. Components pre-define this label for all of their resources. If you add namespaces -n parameter followed by -c parameter, it restricts the collection scope to the namespace of -n parameter. If you don't pass the namespaces, by default, the namespace is set to common-services and foundational-services. By using -c parameter, you cannot collect the MustGather diagnostics for multiple components with single execution. If you want to collect the MustGather diagnostics for different components, run the separate must-gather command with other component name. For more information, see Collecting logs for the resources with same labels. | | -s ex
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Collects MustGather diagnostics information by using custom script that is not the part of a must-gather image. The custom script is placed inside the Docker image that needs to be hosted in IBM trusted registries. To use this feature, pass the component name of -c parameter followed by -s ex parameter. For more information, see Using custom script to collect MustGather diagnostics that are not provided by default. |

Usage examples

The following examples show how to use the MustGather tool to collect specified support data. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

Note: Before you run the code, edit the following parameters in the script:

Collecting support information with default data modules
  1. Run the following command to save the must-gather-default.sh script on your local host.

     cat >must-gather-default.sh <<'EOT'
     #!/bin/bash
     export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
     export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
     export MUST_GATHER_MODULES=overview,system,failure,ocp,cloudpak
     oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
     EOT
    
     chmod +x must-gather-default.sh
    
  2. Run the following command to collect the support data.

     ./must-gather-default.sh
    

    Note: To collect customized diagnostic information about your IBM Cloud Pak and foundational services deployments in your cluster, see Collecting IBM Cloud Pak foundational services information for problem determination.

Collecting only the support information about the IBM Cloud Pak
  1. Run the following command to save the must-gather-cp.sh script on your local host.

     cat >must-gather-cp.sh <<'EOT'
     #!/bin/bash
     export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
     export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
     export MUST_GATHER_MODULES=overview,cloudpak
     oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
     EOT
    
     chmod +x must-gather-cp.sh
    
  2. Run the following command to collect the support data.

     ./must-gather-cp.sh
    
Collecting logs for the resources with same labels

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace
Using custom script to collect MustGather diagnostics that are not provided by default

To collect data that is not provided by default, run the following must-gather command with custom script:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -s ex

For example, if the component name is foo, run the following command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex

From foundational services 3.17 version, you can pass the namespace to the command for custom script and use the namespace inside the custom script.

For example, if the component name is foo, and the component custom script needs to collect diagnostic specific to a namespace, you can run the following must-gather command:

oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex -n namespace
Creating and running the custom scripts

Before you run the custom scripts to collect the MustGather diagnostics, components need to perform the following tasks:

Note: Make sure to refer the Standard adoption guide.

  1. Component creates its own custom scripts.
  2. Component places the custom scripts inside the Docker image that is a part of IBM trusted registries.
  3. Component provides the configmap with the image location.

After you run -c parameter with -s ex parameter, must-gather performs the following tasks:

  1. must-gather fetches the image location from the component in the configmap.
  2. After must-gather gets the image details from the configmap, it uses the image to run a container and copy the custom scripts.
  3. must-gather validates and runs the scripts to collect the logs.

Components must follow the standard adoption guide, which includes the following set of rules:

Custom script standard

When the component creates the custom script, use the following guidelines:

See the following sample script:

oc exec -n openshift-ingress ${pod} -- cat  /lib/haproxy/haproxy.config >${BASE_PATH}/${pod}.haproxy.config &

oc logs -n ${NS} ${POD} --all-containers >${BASE_PATH}/${NS}.${POD}.current.log

See the following sample script that uses the namespace from the must-gather command:

if [ -n "$1" ]; then
ns=$1
oc get pods -n ${ns}
Dockerfile standard

Use the following guidelines for Docker image to include custom scripts:

The following sample is of a Dockerfile:

FROM hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/build-images/ubi8-minimal:latest
RUN microdnf update -y && microdnf install rsync tar -y && microdnf clean all &&\
mkdir /home/custom-scripts && chmod777 /home/custom-scripts
COPY collection-scripts/*<custom-scripts>* /home/custom-scripts
IBM trusted registries

Use only the following trusted registries for Docker image:

Configmap standard

Use the following guidelines when the component provides configmap:

The following sample is of a configmap file with name ibm-mustgather-customscript-<component-name>:

apiVersion: v1
kind: ConfigMap
metadata:
  name: ibm-mustgather-customscript-<component-name>
  namespace: <component specific namespace>
labels:
  serviceability-addon:component-name
data:
  <component-name>: icr.io/cpopen/operator@sha256:6535136...
immutable: true

Note: MustGather also supports default configmap that keeps image location of the component. However, to manage better and avoid any dependencies, MustGather recommends the component to bring its own configmap by following Standard adaption guidelines.

Using MustGather for non-cluster admin

You can use MustGather for non-cluster admin role.

Prerequisites

Note: A non-cluster admin can use all the must-gather commands that an admin uses. However, non-cluster admins must add the --run-namespace parameter to each command. To use the --run-namespace parameter, you need to install Red Hat OpenShift Container Platform 4.12 CLI. The 4.12 CLI also works in older versions of OpenShift clusters.

Permission for non-cluster admin

Login as a cluster admin user and assign the ClusterRole and ClusterRoleBinding roles for non-cluster admin user to collect data for modules, such as Overview, Failure, and Cloudpak.

See the following sample ClusterRole definition:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

See the following sample ClusterRoleBindingdefinition:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

Collecting information for Overview module

To collect information about the Overview module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods" ,"nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

Collecting information for failure module

To collect information about the Failure module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

Collecting information for Cloud pak module

To collect information about the Cloud pak module, create a single YAML file with the following ClusterRole and ClusterRoleBinding definitions. Then, run the oc apply -f <file-name>.yaml command to create the ClusterRole and ClusterRoleBinding resources.

Sample command:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:{Namespace-Name}::default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

Example procedure

The following example includes definitions of all resources that are required to enable non-cluster admins to run the must-gather commands. To create the resources, copy and paste the contents at the CLI or to the YAML editor in your Red Hat OpenShift Container Platform console.

cat <<EOF | tee >(oc apply -f -) | cat >/dev/null
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods", "pods/log", "pods/exec"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cpfs-mg-cluster-rolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: cpfs
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cpfs-mg-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrole
rules:
- apiGroups: [""]
  resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
  resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs"]
  verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
  resources: ["clusterserviceversions"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
  resources: ["*"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-overview-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-overview-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-failure-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-failure-clusterrole

---

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
  resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
  resources: ["ingresses", "networkpolicies"]
  verbs: ["get", "list"]
- apiGroups: [""]
  resources: ["namespaces"]
  verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
  resources: ["clusteroperators", "clusterversions"]
  verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
  resources: ["routes"]
  verbs: ["get", "list"]
- apiGroups: ["batch"]
  resources: ["cronjobs", "jobs"]
  verbs: ["get", "list"]
- apiGroups: ["apps"]
  resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["networks", "imagecontentsourcepolicies"]
  verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
  resources: ["certificatesigningrequests"]
  verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
  resources: ["imagecontentsourcepolicies", "networks"]
  verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
  resources: ["machineconfigpools", "machineconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
  resources: ["volumeattachments"]
  verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
  resources: ["customresourcedefinitions"]
  verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["pods", "nodes"]
  verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
  resources: ["clusterservicestatuses"]
  verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
  resources: ["servicemonitors"]
  verbs: ["get", "list"]
- apiGroups: ["policy"]
  resources: ["endpointslices", "poddisruptionbudgets"]
  verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
  resources: ["endpointslices"]
  verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
  resources: ["horizontalpodautoscalers"]
  verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
  resources: ["buildconfigs", "builds"]
  verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
  resources: ["deploymentconfigs"]
  verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
  resources: ["imagestreams"]
  verbs: ["get", "list"]

---

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: mg-cloudpak-clusterrolebinding
subjects:
  - kind: User
    apiGroup: rbac.authorization.k8s.io
    name: system:serviceaccount:ibm-common-services:default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: mg-cloudpak-clusterrole

EOF

Quick start guide to get support information for non-cluster admin

  1. Navigate to the directory where you want to store the MustGather data.

  2. Collect the data by running the following command.

     oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME}
    

    Where the is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/.

    For example:

     oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n ibm-common-services
    

    The collected data is compressed into a .tgz output file and stored in ./must-gather.local...

  3. Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal.

Usage and examples: non-cluster admin

Available command options for non-cluster admin

You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster for non-cluster admin, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.

The following example shows all available command options:

oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Note: To know more about oc adm must-gather command options and gather command options, see oc adm must-gather command options and gather command options.

Usage examples: non-cluster admin

The following examples show how to use the MustGather tool to collect specified support data for non-cluster admin. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.

oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v

Where NAMESPACE_NAME parameter is the namespace name that belongs to the non-cluster admin and IBM Cloud Pak_NAMESPACES parameter is one or multiple namespaces from where the user collects to mustgather information.

For example, a non-cluster admin is assigned to ibm-common-services namespace and that non-cluster admin wants to collect mustgather logs from different namespaces, such as, ibm-common-services, iaf-demo for the Overview module, then the sample command might resemble the following command:

oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:4.6.14 -- gather -m overview -n ibm-common-services,iaf-demo

Note: Before you run the code, edit the following parameters in the script:

Note: Non-cluster admin supports Overview, Failure and Cloudpak data modules.

Collecting support information with default data modules for non-cluster admin
  1. Run the following command to save the must-gather-default.sh script on your local host.

     cat >must-gather-default.sh <<'EOT'
     #!/bin/bash
     export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
     export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
     export MUST_GATHER_MODULES=overview,failure,cloudpak
     oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
     EOT
    
     chmod +x must-gather-default.sh
    
  2. Run the following command to collect the support data.

     ./must-gather-default.sh
    
Collecting only the support information about the IBM Cloud Pak for non-cluster admin
  1. Run the following command to save the must-gather-cp.sh script on your local host.

     cat >must-gather-cp.sh <<'EOT'
     #!/bin/bash
     export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version>
     export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES>
     export MUST_GATHER_MODULES=overview,cloudpak
     oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES
     EOT
    
     chmod +x must-gather-cp.sh
    
  2. Run the following command to collect the support data.

     ./must-gather-cp.sh
    
Collecting logs for resources with the same labels for non-cluster admin

Prerequisite: Include the label serviceability-addon=<component-name> in all the resources.

To collect the logs only for the resources with the label serviceability-addon=<component-name> in provided namespaces, run the following command:

oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace

Using MustGather diagnostics on a multiarchitecture cluster

In a multiarchitecture cluster, use the must-gather options --node-name or —node-sector to choose non-ARM nodes to run MustGather diagnostics.

See the following example:

oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME} --node-selector 'node-role.kubernetes.io/worker'

The --node-selector argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the --node-name argument to run the packet capture on a single node.