Collecting support information about the cluster
The MustGather tool collects information about your cluster that is crucial for troubleshooting problems and providing support.
The MustGather tool collects support information about the IBM Cloud Paks and the clusters where they are deployed. You can use this tool to gather information that is needed and relevant to the problem before you open a case with the IBM® Support .
The MustGather tool is built on the OpenShift MustGather tool .
- Using MustGather diagnostics through the command line for cluster admin
- Available data modules
- Using MustGather for non-cluster admin
- Using MustGather diagnostics on a multiarchitecture cluster
Using MustGather diagnostics through the command line for cluster admin
Important: IBM Cloud Pak foundational services version 3.18 onwards, the MustGather image for IBM Cloud Paks is available at a new location. If you use the latest
tag for the MustGather image, the image from the new
location is supported on all versions of foundational services.
If you need to use a specific image version, see the following notes:
- For foundational services versions 3.18 and later, use the images from
icr.io/cpopen/cpfs/
. - For foundational services versions 3.17 and prior, use the images from quay.io .
All references and commands in the following sections point to the icr.io/cpopen/cpfs/
location. If your foundational services version is 3.17 or prior, and you need to use a specific image version, you can replace icr.io/cpopen/cpfs/
with quay.io/opencloudio/
, where required.
Available data modules
IMPORTANT: Non-cluster admin supports Overview, Failure and Cloudpak data modules. Cluster admin supports all the available data modules.
To collect a specific type of support information only, define the data modules when you run the MustGather tool. The following table lists the details of all available modules.
Note: The default scan does not collect information about secrets as it is a user sensitive data. To collect information about secrets, you can use the dedicated data module.
Data module | Collected cluster data | Included in the default scan? |
---|---|---|
overview |
|
Yes |
system |
system data module because of the restricted
Security Context Constraints. |
Yes |
failure |
|
Yes |
ocp |
|
Yes |
cloudpak |
|
Yes |
secrets |
|
No |
etcd Available for OpenShift version 4.5 and higher |
|
No |
route |
|
No |
Prerequisites
- Access to the cluster as a user with a cluster-admin role.
- The OpenShift Container Platform CLI (oc) installed. For more information, see Installing the CLI .
-
Access to the MustGather image for IBM Cloud Paks. See the previous section.
Note: For offline environments that cannot access icr.io, the Docker image is available in the offline installation package. For more information, see Downloading the image in offline environments.
Quick start
Complete the following steps to retrieve the support information from your cluster.
- Navigate to the directory where you want to store the MustGather data.
-
Collect the data by running the following command. The
<image_version>
is the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n common-service,<your-foundational-services-namespace>
For example:
oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n common-service,<your-foundational-services-namespace>
The collected data is compressed into a .tgz output file and stored in ./must-gather.local.<rand>
.
- Attach the .tgz output file that contains the support information when you submit the support case on the IBM Support Portal .
Usage and examples- cluster admin
Available command options
You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.
The following example shows all available command options:
oc adm must-gather --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
oc adm must-gather
command options
Option | Description |
---|---|
--image |
A MustGather plugin image to run. Specify the image name and version, for example, icr.io/cpopen/cpfs/must-gather:4.5.16 . |
Note: oc adm must-gather
is a default OpenShift command. Run the oc adm must-gather -h
command for more information.
gather
command options:
Option|Description |---|---| |
-m
| Define the data module that specifies what type of information is collected. For more information, see Available data modules. When this option is not set, the MustGather tool collects information
that includes the following data modules: overview, system, failure, ocp, cloudpak. |
-n
--namespaces=
| Specify the IBM Cloud Pak namespace or namespaces from which the data is collected for the cloudpacks
and secrets
data modules. By default, the namespace is set to foundational-services
.
Separate each namespace with a comma, for example <your-foundational-services-namespace>,common-service
. | |
-h
| Display the help message.| |
-v
| Display the version.| |
-l
| Specify the IBM Cloud Pak labels to collect logs. Separate the labels with a comma. This option requires you to set the -m cloudpak
at the same time. By default, no labels are defined.| |
-c
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Specify the component name to collect logs. If you pass the component-name followed by -c
parameter, must-gather
collects
the MustGather diagnostics with the label serviceability-addon:<component-name>
. Components pre-define this label for all of their resources. If you add namespaces -n
parameter followed by -c
parameter,
it restricts the collection scope to the namespace of -n
parameter. If you don't pass the namespaces, by default, the namespace is set to common-services
and foundational-services
. By using -c
parameter, you cannot collect the MustGather diagnostics for multiple components with single execution. If you want to collect the MustGather diagnostics for different components, run the separate must-gather
command with other component
name. For more information, see Collecting logs for the resources with same labels. | |
-s ex
Available for IBM Cloud Pak foundational services version 3.11.0 and later. | Collects MustGather diagnostics information by using custom script that is not the part of a must-gather
image. The custom script
is placed inside the Docker image that needs to be hosted in IBM trusted registries. To use this feature, pass the component name of -c
parameter followed by -s ex
parameter. For more information, see Using custom script to collect MustGather diagnostics that are not provided by default.
|
Usage examples
The following examples show how to use the MustGather tool to collect specified support data. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.
Note: Before you run the code, edit the following parameters in the script:
- Provide a namespace where your IBM Cloud Pak is deployed in the
CLOUDPAK_NAMESPACES
parameter. - For offline use, set
MUST_GATHER_IMAGE
to the name of your offline MustGather image. - For online use, set the
<image_version>
to the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.
Collecting support information with default data modules
-
Run the following command to save the
must-gather-default.sh
script on your local host.cat >must-gather-default.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,system,failure,ocp,cloudpak oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-default.sh
-
Run the following command to collect the support data.
./must-gather-default.sh
Note: To collect customized diagnostic information about your IBM Cloud Pak and foundational services deployments in your cluster, see Collecting IBM Cloud Pak foundational services information for problem determination.
Collecting only the support information about the IBM Cloud Pak
-
Run the following command to save the
must-gather-cp.sh
script on your local host.cat >must-gather-cp.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,cloudpak oc adm must-gather --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-cp.sh
-
Run the following command to collect the support data.
./must-gather-cp.sh
Collecting logs for the resources with same labels
Prerequisite: Include the label serviceability-addon=<component-name>
in all the resources.
To collect the logs only for the resources with the label serviceability-addon=<component-name>
in provided namespaces, run the following command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace
Using custom script to collect MustGather diagnostics that are not provided by default
To collect data that is not provided by default, run the following must-gather
command with custom script:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -s ex
For example, if the component name is foo
, run the following command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex
From foundational services 3.17 version, you can pass the namespace to the command for custom script and use the namespace inside the custom script.
For example, if the component name is foo
, and the component custom script needs to collect diagnostic specific to a namespace, you can run the following must-gather
command:
oc adm must-gather --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c foo -s ex -n namespace
Creating and running the custom scripts
Before you run the custom scripts to collect the MustGather diagnostics, components need to perform the following tasks:
Note: Make sure to refer the Standard adoption guide.
- Component creates its own custom scripts.
- Component places the custom scripts inside the Docker image that is a part of IBM trusted registries.
- Component provides the configmap with the image location.
After you run -c
parameter with -s ex
parameter, must-gather
performs the following tasks:
must-gather
fetches the image location from the component in the configmap.- After
must-gather
gets the image details from the configmap, it uses the image to run a container and copy the custom scripts. must-gather
validates and runs the scripts to collect the logs.
Components must follow the standard adoption guide, which includes the following set of rules:
Custom script standard
When the component creates the custom script, use the following guidelines:
- By default, the script type is bash or shell scripts.
- If you have more than one script, name the master script as
ibm-mustgather-customscript-<component-name>
. A master script is a script from where the execution starts.- Place the scripts inside the
/home/custom-scripts
folder. - Keep the output path as
${BASE_PATH}
for data collection. - Use only
get
,list
, andexec
commands on the resources.
- Place the scripts inside the
See the following sample script:
oc exec -n openshift-ingress ${pod} -- cat /lib/haproxy/haproxy.config >${BASE_PATH}/${pod}.haproxy.config &
oc logs -n ${NS} ${POD} --all-containers >${BASE_PATH}/${NS}.${POD}.current.log
See the following sample script that uses the namespace from the must-gather
command:
if [ -n "$1" ]; then
ns=$1
oc get pods -n ${ns}
Dockerfile standard
Use the following guidelines for Docker image to include custom scripts:
- Use the standard base image,
ubi8:minimal
. - Install the rsync and tar packages.
- Create a folder
/home/custom-scripts
and copy the scripts in this folder. Give 777 permission to this folder.
The following sample is of a Dockerfile:
FROM hyc-cloud-private-edge-docker-local.artifactory.swg-devops.com/build-images/ubi8-minimal:latest
RUN microdnf update -y && microdnf install rsync tar -y && microdnf clean all &&\
mkdir /home/custom-scripts && chmod777 /home/custom-scripts
COPY collection-scripts/*<custom-scripts>* /home/custom-scripts
IBM trusted registries
Use only the following trusted registries for Docker image:
- quay.io/opencloudio/
- cp.icr.io/cp/icp-foundation/
- cp.stg.icr.io/cp/icp-foundation/
- icr.io/cpopen/
Configmap standard
Use the following guidelines when the component provides configmap:
- Name the configmap as
ibm-mustgather-customscript-<component-name>
and the label asserviceability-addon=<component-name>
. - Set the field
immutable
astrue
in configmap. - Provide the image location as key value pair, where key is the component name, value is the image location with
sha256
manifest. - Don't use image tags. You can use
sha256
manifest.
The following sample is of a configmap file with name ibm-mustgather-customscript-<component-name>
:
apiVersion: v1
kind: ConfigMap
metadata:
name: ibm-mustgather-customscript-<component-name>
namespace: <component specific namespace>
labels:
serviceability-addon:component-name
data:
<component-name>: icr.io/cpopen/operator@sha256:6535136...
immutable: true
Note: MustGather also supports default configmap that keeps image location of the component. However, to manage better and avoid any dependencies, MustGather recommends the component to bring its own configmap by following Standard adaption guidelines.
Using MustGather for non-cluster admin
You can use MustGather for non-cluster admin role.
Prerequisites
-
Access to the cluster as a user with non-cluster admin role. For more information, see Permissions for non-cluster admin.
-
The OpenShift Container Platform version 4.12 CLI is installed. For more information, see Installing the CLI.
-
Access to the MustGather image for IBM Cloud Paks.
Note: A non-cluster admin can use all the must-gather
commands that an admin uses. However, non-cluster admins must add the --run-namespace
parameter to each command. To use the --run-namespace
parameter, you need to install Red Hat OpenShift Container Platform 4.12 CLI. The 4.12 CLI also works in older versions of OpenShift clusters.
Permission for non-cluster admin
Login as a cluster admin user and assign the ClusterRole
and ClusterRoleBinding
roles for non-cluster admin user to collect data for modules, such as Overview, Failure, and Cloudpak.
See the following sample ClusterRole
definition:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
See the following sample ClusterRoleBinding
definition:
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-cluster-rolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: cpfs
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cpfs-mg-clusterrole
Collecting information for Overview module
To collect information about the Overview
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command
to create the ClusterRole
and ClusterRoleBinding
resources.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrole
rules:
- apiGroups: [""]
resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
resources: ["clusterserviceversions"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods" ,"nodes"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
resources: ["*"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-overview-clusterrole
Collecting information for failure module
To collect information about the Failure
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command
to create the ClusterRole
and ClusterRoleBinding
resources.
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}::default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-failure-clusterrole
Collecting information for Cloud pak module
To collect information about the Cloud pak
module, create a single YAML file with the following ClusterRole
and ClusterRoleBinding
definitions. Then, run the oc apply -f <file-name>.yaml
command to create the ClusterRole
and ClusterRoleBinding
resources.
Sample command:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["networks", "imagecontentsourcepolicies"]
verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["imagecontentsourcepolicies", "networks"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools", "machineconfigs"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list"]
- apiGroups: ["policy"]
resources: ["endpointslices", "poddisruptionbudgets"]
verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
resources: ["buildconfigs", "builds"]
verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
resources: ["deploymentconfigs"]
verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
resources: ["imagestreams"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:{Namespace-Name}::default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-cloudpak-clusterrole
Example procedure
The following example includes definitions of all resources that are required to enable non-cluster admins to run the must-gather
commands. To create the resources, copy and paste the contents at the CLI or to the YAML editor in your
Red Hat OpenShift Container Platform console.
cat <<EOF | tee >(oc apply -f -) | cat >/dev/null
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-clusterrole
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/exec"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cpfs-mg-cluster-rolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: cpfs
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cpfs-mg-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrole
rules:
- apiGroups: [""]
resources: ["pods" ,"services", "services/finalizers", "endpoints", "persistentvolumeclaims", "events", "configmaps", "secrets", "nodes", "persistentvolumes", "resourcequotas"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: ["get", "list"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: ["certmanager.k8s.io"]
resources: ["certificates", "challenges", "clusterissuers", "issuers", "orders"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "replicasets", "statefulsets"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs"]
verbs: ["get", "list"]
- apiGroups: ["operators.coreos.com"]
resources: ["clusterserviceversions"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["operator.ibm.com"]
resources: ["*"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-overview-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-overview-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-failure-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-failure-clusterrole
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrole
rules:
- apiGroups: [""]
resources: ["persistentvolumes", "persistentvolumeclaims", "services", "endpoints", "nodes", "secrets", "events", "configmaps", "replicationcontrollers"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods/log"]
verbs: ["get", "list", "create"]
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses", "networkpolicies"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: ["config.openshift.io"]
resources: ["clusteroperators", "clusterversions"]
verbs: ["get", "list"]
- apiGroups: ["route.openshift.io"]
resources: ["routes"]
verbs: ["get", "list"]
- apiGroups: ["batch"]
resources: ["cronjobs", "jobs"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "daemonsets", "statefulsets", "replicasets"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["networks", "imagecontentsourcepolicies"]
verbs: ["get", "list"]
- apiGroups: ["certificates.k8s.io"]
resources: ["certificatesigningrequests"]
verbs: ["get", "list"]
- apiGroups: ["operator.openshift.io"]
resources: ["imagecontentsourcepolicies", "networks"]
verbs: ["get", "list"]
- apiGroups: ["machineconfiguration.openshift.io"]
resources: ["machineconfigpools", "machineconfigs"]
verbs: ["get", "list"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["get", "list"]
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list"]
- apiGroups: ["clusterhealth.ibm.com"]
resources: ["clusterservicestatuses"]
verbs: ["get", "list"]
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list"]
- apiGroups: ["policy"]
resources: ["endpointslices", "poddisruptionbudgets"]
verbs: ["get", "list"]
- apiGroups: ["discovery.k8s.io"]
resources: ["endpointslices"]
verbs: ["get", "list"]
- apiGroups: ["autoscaling"]
resources: ["horizontalpodautoscalers"]
verbs: ["get", "list"]
- apiGroups: ["build.openshift.io"]
resources: ["buildconfigs", "builds"]
verbs: ["get", "list"]
- apiGroups: ["apps.openshift.io"]
resources: ["deploymentconfigs"]
verbs: ["get", "list"]
- apiGroups: ["image.openshift.io"]
resources: ["imagestreams"]
verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: mg-cloudpak-clusterrolebinding
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: system:serviceaccount:ibm-common-services:default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: mg-cloudpak-clusterrole
EOF
Quick start guide to get support information for non-cluster admin
-
Navigate to the directory where you want to store the MustGather data.
-
Collect the data by running the following command.
oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME}
Where the
is the version of the MustGather image that you can find at icr.io/cpopen/cpfs/. For example:
oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -n ibm-common-services
The collected data is compressed into a
.tgz
output file and stored in./must-gather.local..
. -
Attach the
.tgz
output file that contains the support information when you submit the support case on the IBM Support Portal.
Usage and examples: non-cluster admin
Available command options for non-cluster admin
You can use the MustGather tool to collect support information about IBM Cloud Paks. You can collect all support information from your cluster for non-cluster admin, or only specified information about specified namespaces. Add options to the MustGather command to collect the data you need.
The following example shows all available command options:
oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
Note: To know more about oc adm must-gather
command options and gather
command options, see oc adm must-gather
command options and gather
command options.
Usage examples: non-cluster admin
The following examples show how to use the MustGather tool to collect specified support data for non-cluster admin. When you copy and run the code that is provided in the examples, the shell script is saved on the local host. You can reuse the script anytime to collect the support information.
oc adm must-gather --run-namespace=<NAMESPACE_NAME> --image=<MUST_GATHER_IMAGE> -- gather -m <MUST_GATHER_MODULES> -n <IBM Cloud Pak_NAMESPACES> | -h | -v
Where NAMESPACE_NAME parameter is the namespace name that belongs to the non-cluster admin and IBM Cloud Pak_NAMESPACES parameter is one or multiple namespaces from where the user collects to mustgather information.
For example, a non-cluster admin is assigned to ibm-common-services
namespace and that non-cluster admin wants to collect mustgather logs from different namespaces, such as, ibm-common-services
, iaf-demo
for
the Overview
module, then the sample command might resemble the following command:
oc adm must-gather --run-namespace=ibm-common-services --image=icr.io/cpopen/cpfs/must-gather:4.6.14 -- gather -m overview -n ibm-common-services,iaf-demo
Note: Before you run the code, edit the following parameters in the script:
- Provide a namespace where your IBM Cloud Pak is deployed in the
CLOUDPAK_NAMESPACES
parameter. - For offline use, set
MUST_GATHER_IMAGE
to the name of your offline MustGather image. - For online use, set the
<image_version>
to the version of the MustGather image that you can find aticr.io/cpopen/cpfs/
.
Note: Non-cluster admin supports Overview, Failure and Cloudpak data modules.
Collecting support information with default data modules for non-cluster admin
-
Run the following command to save the
must-gather-default.sh
script on your local host.cat >must-gather-default.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,failure,cloudpak oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-default.sh
-
Run the following command to collect the support data.
./must-gather-default.sh
Collecting only the support information about the IBM Cloud Pak for non-cluster admin
-
Run the following command to save the
must-gather-cp.sh
script on your local host.cat >must-gather-cp.sh <<'EOT' #!/bin/bash export MUST_GATHER_IMAGE=icr.io/cpopen/cpfs/must-gather:<image_version> export CLOUDPAK_NAMESPACES=common-service,ibm-common-services,<CLOUDPAK_NAMESPACES> export MUST_GATHER_MODULES=overview,cloudpak oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=$MUST_GATHER_IMAGE -- gather -m $MUST_GATHER_MODULES -n $CLOUDPAK_NAMESPACES EOT chmod +x must-gather-cp.sh
-
Run the following command to collect the support data.
./must-gather-cp.sh
Collecting logs for resources with the same labels for non-cluster admin
Prerequisite: Include the label serviceability-addon=<component-name>
in all the resources.
To collect the logs only for the resources with the label serviceability-addon=<component-name>
in provided namespaces, run the following command:
oc adm must-gather --run-namespace={NAMESPACE-NAME} --image=icr.io/cpopen/cpfs/ibm-must-gather:latest -- gather -c <component name> -n namespace
Using MustGather diagnostics on a multiarchitecture cluster
In a multiarchitecture cluster, use the must-gather
options --node-name
or —node-sector
to choose non-ARM nodes to run MustGather diagnostics.
See the following example:
oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:<image_version> -- gather -n {NAMESPACE-NAME} --node-selector 'node-role.kubernetes.io/worker'
The --node-selector
argument and example value specifies to perform the packet captures on the worker nodes. As an alternative, you can specify the --node-name
argument to run the packet capture on a single node.