Troubleshooting

The must-gather tool gathers information about IBM Cloud Pak® for Integration and the cluster where it is deployed. The information that is collected is required by IBM Support for troubleshooting user issues.

Before you open a case with IBM Support, get the logs and other important information by using the must-gather tool.

Before you begin

Meet the following prerequisites:

Gathering Cloud Pak for Integration information

Running the must-gather tool as an administrator

When you run the must-gather tool with admin permissions, the tool automatically finds namespaces where instances are installed and collects all the required logs. It also collect logs from any namespace with the prefix openshift- in the name.

If you are installing Cloud Pak for Integration in an air-gapped or disconnected environment, skip to the "Air-gapped and disconnected environments" section.

  1. Log in to the cluster:

    oc login
  2. Go to the directory where you want to store the must-gather data.

  3. Collect the must-gather information. Make sure you include the quotation marks in the command.

    • To collect Cloud Pak for Integration must-gather only, run:

      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c cp4i'"
    • To collect Cloud Pak for Integration must-gather with a version table, run:

      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c cp4i -v True'"
    • To get additional options (help), run:

      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-h'"
      
      options:
      -h, --help          Show this help message and exit
      -n NAMESPACE, --namespace NAMESPACE
                          Namespace you want to collect the logs for, only works with -c appconnect
      -p PODNAME, --podname PODNAME
                          Pod name you want to collect logs for, only works with -c appconnect
      -c COMPONENT, --component COMPONENT
                          Component you want to collect logs for. For ACE = appconnect, CP4I = cp4i
      -v VERSION, --version VERSION
                          To get a table of all the versions installed, set -v True
      --previous_pod_logs, PREVIOUS POD LOGS
                          To collect pod logs for terminated pods, set --previous_pod_logs True 
      -ne, --namespaced_events NAMESPACED EVENTS
                          To collect events per namespace, set -ne True                         

Running the must-gather tool as a non-administrator user

If you do not have admin permissions, you can collect logs only from the namespaces where you have user permissions.

  1. Log in to the cluster:

    oc login
  2. Run the following command to create a directory called mustgather (where you will store the logs that are collected) and to apply permissions so that logs can be written to the directory. This directory is mounted when you run the command for the must-gather tool in the last step.

    mkdir -p mustgather
    chmod 777 mustgather
    
    Change the dir to mustgather folder
    
    cd mustgather
  3. Collect the logs. For the value of namespaces, enter a list of comma-separated namespaces (to which you have permissions) that you want to collect logs from.

    image_name=icr.io/cpopen/cpfs/must-gather:latest
    
    namespaces=<my_namespaces>

    For example:

    namespaces=navigator-ns,ibm-common-services,apic-ns,ace-dashboard-ns,ace-integrationserver-ns

    Now run the must-gather tool. Make sure that you include the quotation marks in the command.

    podman run --platform linux/amd64  --pull=always -v ${KUBECONFIG:-~/.kube/config}:/kube/config -v $(pwd):/tmp/must-gather \
    --env MYENV=NONE --env KUBECONFIG=/kube/config \
    --env MUSTGATHERPATH=/tmp/must-gather \
    ${image_name} gather -m cp4i --params "-c cp4i --non_admin True -n ${namespaces}"

Air-gapped and disconnected environments

This section applies to installations in air-gapped and disconnected environments only.

You can use the must-gather tool in an air-gapped environment because when you mirrored the Cloud Pak foundational services images (using the procedure in Mirroring images with a bastion host or Mirroring images with a portable compute or storage device (file system)), the must-gather image was mirrored to your local registry.

  1. Set the registry where your local images are mirrored:

    export LOCAL_REGISTRY=<YOUR_LOCAL_REPOSITORY>
  2. Collect the must-gather information. Make sure you include the quotation marks in the command.

    • To collect Cloud Pak for Integration must-gather only, run:

      oc adm must-gather --image=${LOCAL_REGISTRY}/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c cp4i'"
    • To collect Cloud Pak for Integration must-gather with a version table, run:

      oc adm must-gather --image=${LOCAL_REGISTRY}/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c cp4i -v True'"
    • To get additional options (help), run:

      oc adm must-gather --image=${LOCAL_REGISTRY}/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-h'"
      
      options:
      -h, --help         Show this help message and exit
      -n NAMESPACE, --namespace NAMESPACE
                         Namespace you want to collect the logs for, only works with -c appconnect
      -p PODNAME, --podname PODNAME
                         Pod name you want to collect logs for, only works with -c appconnect
      -c COMPONENT, --component COMPONENT
                         Component you want to collect logs for. For ACE = appconnect, CP4I = cp4i
      -v VERSION, --version VERSION
                         To get a table of all the versions installed, set -v True
       --previous_pod_logs, PREVIOUS POD LOGS
                           To collect pod logs for terminated pods, set --previous_pod_logs True 
       -ne, --namespaced_events NAMESPACED EVENTS
                           To collect events per namespace, set -ne True             
      
    • To confirm that you have mirrored the latest available version of the must-gather tool, run:

      skopeo list-tags docker://[LOCAL_REGISTRY:5000]/cpopen/cpfs/must-gather
      {
          "Repository": "${LOCAL_REGISTRY}/cpopen/cpfs/must-gather",
          "Tags": [
              "4.5.16"
              .....
              "4.6.7",
              "4.6.8",
              "4.6.9",
              "latest"
          ]
       }

Information that is collected

The must-gather tool collects information about the following resources:

Certificate
Clients
ConfigMaps
Cp4iServiceBindings
Cloud Pak CustomResources
Deployments
InstallPlan
Jobs
OperandRequests
PersistentVolumeClaims
Pod-Description
Pod-Logs
ReplicaSets
Routes
Services
StatefulSets
Subscriptions
CatalogSources
ServiceAccounts
Events
OperandBindInfo
OperandRegistries
OperandConfigs
Issuer

It collects information from any namespace where operators and instances are installed, as well as the following namespaces:

ibm-common-services
openshift-marketplace
openshift-operator-lifecycle-manager
openshift-operators
openshift-ingress
kube-public

The must-gather tool collects only the name of the secrets from the listed namespaces. It doesn't collect the data in those secrets.

Gathering Cloud Pak foundational services logs only

The script in the previous section, Gathering extended Cloud Pak For Integration information, also returns the pod logs for Cloud Pak foundational services.

However, if you need to work directly with the Cloud Pak foundational services support team, see Collecting Cloud Pak Cluster and Common services for problem determination in the Support documentation. This page describes how to collect logs that are needed by the team that investigates your issue.

Gathering OpenShift Container Platform logs

OpenShift has its own must-gather tool. To gather debugging information about your cluster, see Gathering data about your cluster).

Known limitations and workarounds

To get specific information about troubleshooting known issues in Cloud Pak for Integration, see Known limitations.