Gathering diagnostic information

Before you raise a request for support, gather diagnostic information that you can send to IBM® Support to help us investigate your issue. You can use the MustGather tool for IBM Cloud Paks to gather diagnostic information from your Red Hat® OpenShift® or Kubernetes cluster.

The MustGather tool is built on the Red Hat OpenShift must-gather tool, which is typically run with the oc adm must-gather command. For more information about the Red Hat OpenShift must-gather tool, see Gathering data about your cluster in the Red Hat OpenShift documentation.

To use the MustGather tool for IBM Cloud Paks, you will need to access the MustGather image for IBM Cloud Paks. An entitlement key is not required to pull this image. For more information, see Using MustGather diagnostics through the command line for cluster admin.

In the documented commands, docker and podman can be used interchangeably.

Gathering diagnostic information from a Red Hat OpenShift environment

If you are running the IBM App Connect Operator in an IBM Cloud Pak for Integration deployment or as an independent deployment on Red Hat OpenShift, you can use the MustGather tool to gather diagnostic information from your cluster.

About this task

If you want to run the MustGather tool as an administrator, ensure that you have cluster administrator authority with cluster-admin permissions.

Complete either of the following tasks to gather diagnostic information:

Running the MustGather tool as an administrator

When you run the MustGather tool with admin permissions, the tool automatically finds namespaces where instances are installed and collects all the required information. You can alternatively use the tool to collect information from a specific namespace and all pods, or a specific pod, within that namespace.

Procedure

To gather and submit diagnostic information that is specific to App Connect, complete the following steps:

  1. From the command line, log in to your Red Hat OpenShift cluster by using the oc login command.
  2. Navigate to a temporary directory where you want to store the diagnostic information that the MustGather tool collects.
  3. Collect the information by completing one of these steps:
    • To collect diagnostic information (including version information) from all the namespaces where the IBM App Connect Operator is installed, run the following command. The -v True setting provides a version table that lists the operand and Operator versions for all deployed resources (for example, DesignerAuthoring, Dashboard, IntegrationServer, IntegrationRuntime, and Configuration) in all the namespaces.
      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c appconnect -v True'"
    • To collect diagnostic information (including version information) from a specific namespace, run the following command:
      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c appconnect -v True -n namespaceName'"

      For example, the following command collects information from a namespace named ace:

      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c appconnect -v True -n ace'"
    • To collect diagnostic information (including version information) from a specific namespace and a specific pod within that namespace, run the following command:
      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c appconnect -v True -n namespaceName -p podName'"
      Tip: You can run the oc get pods command to obtain the list of pods in the namespace.

      For example, the following command collects information from a namespace named ace and an integration runtime pod named quickstart-ir-6f9f88cd99-kp6bf within that namespace:

      oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-c appconnect -v True -n ace -p quickstart-ir-6f9f88cd99-kp6bf'"
    Tip: You can obtain help information about the MustGather tool's parameters by running the following command:
    oc adm must-gather --image=icr.io/cpopen/cpfs/must-gather:latest -- gather -m cp4i --params "'-h'"

    For each namespace where the Operator is installed, the MustGather tool captures App Connect-specific information to an archive file in the temporary directory that you ran the oc adm must-gather command from. This file is saved with the name ACE_Data_Collector_*.tar.gz under the following directory path:

    must-gather.local.uniqueID/icr-io-cpopen-cpfs-must-gather-sha256-uniqueID/CloudpakForIntegration-timestamp/namespaceName/ACE_product_logs/podName/ACE_Data_Collector_*.tar.gz

    For example:

    must-gather.local.7871852174993092609/icr-io-cpopen-cpfs-must-gather-sha256-0b2d1aeda13297b153735ade2caa1c3039bcdf9a371f43e8afa4ddfb9d435075/CloudpakForIntegration-2023-11-01/ace/ACE_product_logs/customerdb-01-ir-6cbd4d575-kl5j8/ACE_Data_Collector_20231030-103111_IS.tar.gz

    The full set of subdirectories under must-gather.local.uniqueID/icr-io-cpopen-cpfs-must-gather-sha256-uniqueID/CloudpakForIntegration-timestamp/namespaceName is as follows:

    ACE_product_logs
    ClusterServiceVersion
    ConfigMaps
    CustomResources
    Deployments
    InstallPlan
    OperandRequests
    Pod-Description
    Pod-Logs
    ReplicaSets
    Routes
    ServiceAccounts
    Services
    Subscriptions
  4. Compress the entire contents of the must-gather.local.uniqueID directory to an archive file.
  5. Submit the archive file with your support case as described in Contacting IBM.

Running the MustGather tool as a non-administrator user

If you do not have admin permissions, you can collect diagnostic information only from the namespaces where you have user permissions.

Procedure

To gather and submit diagnostic information that is specific to App Connect, complete the following steps:

  1. From the command line, log in to your Red Hat OpenShift cluster as a non-administrator user by using the oc login command.
  2. Run the following commands to create a temporary directory called mustgather (where you will store the logs that are collected) and to apply permissions so that logs can be written to the directory. Then, switch to the mustgather directory. This directory is mounted when you run the command for the MustGather tool in a later step.
    mkdir -p mustgather
    chmod 777 mustgather
    cd mustgather
  3. Declare the location of the MustGather image for IBM Cloud Paks and the namespaces that you want to collect logs from. For the namespaces value, enter a list of comma-separated namespaces that you have permissions to.
    image_name=icr.io/cpopen/cpfs/must-gather:latest
    
    namespaces=myNamespaces

    For example:

    namespaces=navigator-ns,ibm-common-services,apic-ns,ace-dashboard-ns,ace-integrationserver-ns
  4. To use the MustGather tool to collect diagnostic information from all pods in the specified namespaces, run the following command:
    docker run --platform linux/amd64  --pull=always -v ${KUBECONFIG:-~/.kube/config}:/kube/config -v $(pwd):/tmp/must-gather \
    --env MYENV=NONE --env KUBECONFIG=/kube/config \
    --env MUSTGATHERPATH=/tmp/must-gather \
    ${image_name} gather -m cp4i --params "-c appconnect --non_admin True -n ${namespaces}"
  5. Submit an archive file that contains your diagnostic information with your support case as described in Contacting IBM.

Gathering diagnostic information from a Kubernetes environment

If you are running the IBM App Connect Operator as an independent deployment in a Kubernetes environment, you can use the MustGather tool to gather diagnostic information from your cluster. You can run the tool as a non-administrator user to collect information from the namespaces where you have user permissions.

Procedure

To gather and submit diagnostic information that is specific to App Connect, complete the following steps:

  1. If necessary, log in to your Kubernetes cluster from the command line as a non-administrator user.
  2. Run the following commands to create a temporary directory called mustgather (where you will store the logs that are collected) and to apply permissions so that logs can be written to the directory. Then, switch to the mustgather directory. This directory is mounted when you run the command for the MustGather tool in a later step.
    mkdir -p mustgather
    chmod 777 mustgather
    cd mustgather
  3. Declare the location of the MustGather image for IBM Cloud Paks and the namespaces that you want to collect logs from. For the namespaces value, enter a list of comma-separated namespaces that you have permissions to.
    image_name=icr.io/cpopen/cpfs/must-gather:latest
    
    namespaces=myNamespaces

    For example:

    namespaces=ace-dashboard-ns,ace-integrationserver-ns
  4. To use the MustGather tool to collect diagnostic information from all pods in the specified namespaces, run the following command:
    docker run \
      --platform linux/amd64 \
      --pull=always \
       -v ${KUBECONFIG:-~/.kube/config}:/kube/config \
       -v $(pwd):/tmp/must-gather \
       --env MYENV=NONE \
       --env KUBECONFIG=/kube/config \
       --env MUSTGATHERPATH=/tmp/must-gather \
       ${image_name} gather "-c appconnect --non_admin True -n ${namespaces}"

    Applicable if using an IBM Cloud Kubernetes Service cluster:

    If you logged in to your IBM Cloud Kubernetes Service cluster by using the ibmcloud login command and updated your kube config by using ibmcloud ks, a line like this, which references a .pem file, is typically added to your kube config:

    certificate-authority: /root/.bluemix/plugins/container-service/clusters/acecc-mac-ckna5nnl0j8b50ialk90/ca-aaa00-acecc-mac.pem

    The command that runs the MustGather tool attempts to mount the kube config into the container, but this command will fail because the referenced .pem file will not exist inside the container. To resolve this issue, flatten the kube config before you run the MustGather tool.

    1. Flatten the kube config as follows:
      kubectl config view --flatten --output yaml > ~/.kube/config-flattened
      Flattening a kube config results in references to files being replaced by the content of those files; for example:
      certificate-authority: <REDACTED-PEM-FILE-CONTENTS>
    2. To use the MustGather tool to collect diagnostic information from all pods in the specified namespaces, run the following command:
      docker run \
        --platform linux/amd64 \
        --pull=always \
         -v ${KUBECONFIG:-~/.kube/config-flattened}:/kube/config \
         -v $(pwd):/tmp/must-gather \
         --env MYENV=NONE \
         --env KUBECONFIG=/kube/config \
         --env MUSTGATHERPATH=/tmp/must-gather \
         ${image_name} gather "-c appconnect --non_admin True -n ${namespaces}"

  5. Submit an archive file that contains your diagnostic information with your support case as described in Contacting IBM.