Troubleshooting

It is necessary to collect specific information about your deployments to measure their well-being and to diagnose a problem before you open a support case on Cloud Pak for Business Automation.

Collect specific data about your environment and your Cloud Pak installation before you contact IBM support for assistance with a Cloud Pak for Business Automation issue. Always provide a detailed description of the problem and your environment.

When you run diagnostic commands, run them from an empty directory to package the files more cleanly. Run the commands from the namespace in which you observe the problematic container or component. For more information, see Mustgather: Collecting data to diagnose issues.

Collecting data

If you see any issues during the upgrade, collect the following information that might help with the troubleshooting.
  • Collect the following information from the scripts directory.
    • <location of cert-kubernetes>/scripts/cp4a-script-logs
    • <location of cert-kubernetes>/scripts/cpfs/installer_scripts/cp3pt0-deployment/logs
    • <location of cert-kubernetes>/scripts/cp4ba-upgrade
    • <location of cert-kubernetes>/scripts/.tmp
    • <location of cert-kubernetes>/scripts/*.log
    • <location of cert-kubernetes>/scripts/*.yaml
  • Collect the output from the console where you run the upgrade script.
  • Screenshot of the error messages if applicable.
  • Run the Cloud Pak for Business Automation Mustgather.

Resolving issues

MongoDB migration fails when you upgrade from 21.0.3-IF031 or later to 24.0.0 or interim fixes
During the upgrade of MongoDB, you might see the following error.
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     10s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     20s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     31s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     41s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     52s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     62s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     73s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     83s
[INFO] Waiting for MongoDB copy to initialize
icp-mongodb-0   0/1   Init:1/2   0     94s
[INFO] Waiting for MongoDB copy to initialize
Error from server (NotFound): pods "icp-mongodb-0" not found
[✘] Error occurred in function deploymongocopy at line 1842

[✗] Failed to execute command: /root/cert-kube-2400/cert-kubernetes/scripts/cpfs/installer_scripts/preload_data.sh --original-cs-ns ibm-common-services --services-ns test-ns --yq "/root/cert-kube-2400/cert-kubernetes/scripts/cpfs/yq/amd64/yq"
[ATTENTION]: You can run follow command to try upgrade again after fix migration issue of IBM Cloud Pak foundational services.
           # ./cp4a-deployment.sh -m upgradeOperator -n combank1 --cpfs-upgrade-mode <migration mode> --original-cp4ba-csv-ver <cp4ba-csv-version-before-upgrade>(B[m
           Usage:
           --cpfs-upgrade-mode     : The migration mode for IBM Cloud Pak foundational services, the valid values [shared2shared/shared2dedicated/dedicated2dedicated]
           --original-cp4ba-csv-ver: The version of csv for CP4BA operator before upgrade, the example value [21.3.31] for 21.0.3-IF031/[22.2.6] for 22.0.2-IF006/[23.2.3] for 23.0.2-IF006
           Example command: 
           # ./cp4a-deployment.sh -m upgradeOperator -n test-ns --cpfs-upgrade-mode dedicated2dedicated --original-cp4ba-csv-ver 21.3.31
Solution: Run the preload_data.sh script with the --clean option. For more information, see Preload data. After you complete running the preload_data.sh, run the command that is listed in the upgrade script output.
./cp4a-deployment.sh -m upgradeOperator -n test-ns --cpfs-upgrade-mode dedicated2dedicated --original-cp4ba-csv-ver 21.3.31
Upgrade blocked due to missing ibm-cp4ba-common-config ConfigMap

If the Not found "ibm-cp4ba-common-config" configMap error message is encountered when the upgrade script runs, the ibm-cp4ba-common-config ConfigMap is missing from the namespace for the CP4BA deployment.

To resolve the issue, follow the steps in either Option 1 or Option 2.

Option 1: Create the missing ConfigMap manually

Use the following steps to re-create the ConfigMap.

  1. Find the namespace where the CommonService CR is installed by running the following command. The namespace of the CommonService deployment is where all the Cloud Pak foundational services and CP4BA operators are installed.
    oc get CommonService -A
    Note: If you have multiple deployments, make sure that you locate the namespace that you are upgrading before you complete the next steps.
  2. Run the following commands to retrieve the values needed to create the ibm-cp4ba-common-config ConfigMap.
    oc get CommonService -o yaml | grep 'operatorNamespace:' | head -n 1 | awk '{print $2}'
    oc get CommonService -o yaml | grep 'servicesNamespace:' | head -n 1 | awk '{print $2}'
  3. Create the ibm-cp4ba-common-config ConfigMap in the target CommonService namespace.
    kind: ConfigMap
    apiVersion: v1
    metadata:
      name: ibm-cp4ba-common-config
      namespace: <namespace>
    data:
      operators_namespace: "<operator-namespace>"
      services_namespace: "<service-namespace>"

    Where the value of <namespace> and <operator-namespace> is provided by the first command in the previous step. The value of <service-namespace> is provided by the second command.

  4. If you created a YAML file in the previous step, for example ibm-cp4ba-common-config.yaml, then run the following command to apply the ConfigMap to the CommonService namespace from the command window.
    oc apply -f ibm-cp4ba-common-config.yaml -n <namespace>

    If you prefer to use the OCP console, then apply the ConfigMap manually.

  5. After you created the ibm-cp4ba-common-config ConfigMap, return to upgrading the deployment by running the cp4a-deployment.sh script in upgradeOperator mode.

Option 2: Run the cp4a-clusteradmin-setup.sh script to create the missing ConfigMap

  1. Run the cp4a-clusteradmin-setup.sh script from the local directory where you downloaded the cert-kubernetes repository, and follow the prompts in the command window.
    cd cert-kubernetes/scripts
    ./cp4a-clusteradmin-setup.sh -fix_configmap
  2. After you created the ibm-cp4ba-common-config ConfigMap, return to upgrading the deployment by running the cp4a-deployment.sh script in upgradeOperator mode.

Troubleshooting capabilities

The custom resource can be configured to enable and disable specific logging parameters, log levels, log formats, and where these logs are stored for the various capabilities. If you need more information about specific Cloud Pak capabilities, go to the relevant troubleshooting topics.