Known issues and limitations for IBM Cloud Pak for Data

Namespace deleted when the --namespace parameter is used with the health network-performance command

Applies to: 4.8.4 and 4.8.5

Fixed in: 4.8.5 (v13.1.5r1) and later

To use the --namespace parameter, you must download and reinstall the latest version of cpd-cli (v13.1.5r1). For more information, see Installing the Cloud Pak for Data command-line interface (cpd-cli).

After rebooting the cluster, some services in Cloud Pak for Data on OpenShift Data Foundation aren't functional

Applies to: 4.8.3 and later

Diagnosing the problem

After rebooting the cluster, some Cloud Pak for Data custom resources remain in the InProgress state.

For more information about this problem, see Missing NodeStageVolume RPC call blocks new pods from going into Running state in the Red Hat® OpenShift Data Foundation 4.1.4 release notes.

Workaround

Do the following steps:

1. Find the nodes that have pods that are in an Error state:

   oc get po -n ${PROJECT_CPD_INST_OPERANDS} -o wide s -A  | grep -v -P "Completed|(\d+)\/\1"

2. Mark each node as unschedulable.

   oc adm cordon <node_name>

3. Delete the affected pods:

   oc get pod   | grep -Ev "Comp|0/0|1/1|2/2|3/3|4/4|5/5|6/6|7/7"|awk '{print $1}' |xargs oc delete po --force=true --grace-period=0

4. Mark each node as scheduled:

   oc adm uncordon <node_name>

You cannot activate notification forwarding from the Events tab

Applies to: 4.8.0

Fixed in: 4.8.1

The services catalog includes two IBM Knowledge Catalog tiles

Applies to: 4.8.0 and later

When you go to the Services catalog page in the web client, you see two tiles for the IBM Knowledge Catalog service.

Resolving the problem

To remove the extra tile from the services catalog, restart the zen-watcher pod:

oc delete pods -n=${PROJECT_CPD_INST_OPERANDS} | grep zen-watcher

Some text in services catalog is outdated after upgrade

Applies to: 4.8.0 and 4.8.1

Fixed in: 4.8.2

When you go to the Services catalog page in the web client, you see outdated information after you upgrade Cloud Pak for Data. For example, Watson Assistant was renamed to watsonx Assistant in Cloud Pak for Data Version 4.8. However, the services catalog still displays Watson Assistant.

Resolving the problem

To update the text in the services catalog,

1. Restart the zen-core-api pods:

   oc rollout restart deployment zen-core-api -n=${PROJECT_CPD_INST_OPERANDS}

2. Clear your browser cache and re-authenticate to the web client.

The Identity and user access card on the home page displays an error

Applies to: 4.8.0 and 4.8.1

Fixed in: 4.8.2

The data cannot be displayed.
The service might be restarting. Wait a few minutes and refresh the page. 
If the problem persists, contact your administrator.

However, refreshing the page does not resolve the problem.

Resolving the problem

To resolve the error, clear your web browser cache and cookies.

You are occasionally redirected to the wrong login page when Cloud Pak for Data is integrated with the Identity Management Service

Applies to: 4.8.0 and 4.8.1

Fixed in: 4.8.2

If your session expires and you click the Log in button on the Logout page, you are sometimes redirected to a URL that starts with https://cp-console rather than https://cpd. If you log in to the https://cp-console URL, you are directed to the Identity providers page, which you might not have access to.

Resolving the issue

Use the login URL provided by your administrator, or edit the URL to replace cp-console with cpd and try to log in again.

Intermittent login issues when Cloud Pak for Data is integrated with the Identity Management Service

Applies to: 4.8.0

Some users might be directed to the Identity providers page rather than the Cloud Pak for Data home page.

Diagnosing the problem

If users experience one or more of the issues described in the preceding text, check the platform-identity-provider pods to determine whether the pods have been restarted multiple times:

oc get pods -n ${PROJECT_CPD_INST_OPERANDS} | grep platform-identity-provider

If the output indicates multiple restarts, proceed to Resolving the problem

Resolving the problem

1. Restart the icp-mongodb-0 pod:

   oc delete pods icp-mongodb-0 -n ${PROJECT_CPD_INST_OPERANDS}

2. Restart the icp-mongodb-1 pod:

   oc delete pods icp-mongodb-1 -n ${PROJECT_CPD_INST_OPERANDS}

3. Restart the icp-mongodb-2 pod:

   oc delete pods icp-mongodb-2 -n ${PROJECT_CPD_INST_OPERANDS}

4. Restart the platform-auth-service pod:

   oc delete pods -n ${PROJECT_CPD_INST_OPERANDS} | grep platform-auth-service

5. Restart the platform-identity-management pod:

   oc delete pods -n ${PROJECT_CPD_INST_OPERANDS} | grep platform-identity-management

Warning alerts might appear on the home page after installation

Applies to: 4.8.0 and later

Diagnosing the problem

After installation, you might see warning alerts on the home page. However, the events that generated the alerts have cleared. The alerts will continue to display on the Alerts card for up to 3 days unless you delete the pods that generated the alerts.

The alerts are visible to users with one of the following permissions:

• Administer platform
• Manage platform health
• View platform health

1. Log in to the web client as a user with the appropriate permissions to view alerts.
2. On the home page, click View all on the Alerts card.
3. On the Alerts and events page, confirm that the alerts were generated by one of the following services:
   • Common core services
   • IBM Knowledge Catalog

This issue can occur because wkc-db2-init pods or jdbc-driver-sync-job pods are in an Error state.

Resolving the problem

An instance administrator or cluster administrator must resolve the problem.

1. Log in to Red Hat OpenShift Container Platform as a user with sufficient permissions to complete the task.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Check the status of the wkc-db2-init pods and the jdbc-driver-sync-job pods.

   oc get pods --sort-by=.status.startTime -n ${PROJECT_CPD_INST_OPERANDS} | grep -E 'wkc-db2u-init|jdbc-driver'

3. Delete any pods that are in the Error state.

   Replace <pod-name> with the name of the pod in the error state.

   oc delete pod <pod-name> -n ${PROJECT_CPD_INST_OPERANDS}

Tethered projects are not retrieved

Applied to: 4.8.0

Resolving the problem

Patch the ZenService custom resource to trigger a reconcile loop:

oc patch ZenService lite-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge \
--patch='{"spec": {"patchProductConfigmap": "true"}}'

RSI patches are not applied to pods as expected

Applies to: 4.8.0 and 4.8.1

Fixed in: 4.8.2

In some situations, RSI patches are not applied to a subset of the specified pods. This typically occurs when you use the --select_all_pods option when you create an RSI patch.

Diagnosing the problem

Check the status of the pod owners, such as StatefulSets, to determine whether the owners are stuck in the patch in progress state:

1. Check for any Deployments in the patch in progress state:

   oc get deployment \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any Deployments in this state.

2. Check for any StatefulSets in the patch in progress state:

   oc get statefulset \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any StatefulSets in this state.

3. Check for any ReplicaSets in the patch in progress state:

   oc get replicaset \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any ReplicaSets in this state.

4. Check for any ReplicaControllers in the patch in progress state:

   oc get replicacontroller \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any ReplicaControllers in this state.

5. Check for any Jobs in the patch in progress state:

   oc get jobs \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any Jobs in this state.

6. Check for any CronJobs in the patch in progress state:

   oc get cronjob \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -o=jsonpath='{.items[?(@.metadata.annotations.resourcespecinjector\.ibm\.com/injection_status=="patch in progress")].metadata.name}'

   The command returns a list of any CronJobs in this state.

Resolving the problem

For each resource returned by the preceding commands, update the annotation on the resource:

oc annotate <resource-type> <resource-name> \
-n=${PROJECT_CPD_INST_OPERANDS} \
resourcespecinjector.ibm.com/injection_status-

Replace <resource-type> with the type of the resource and <resource-name> with the name of the resource.

Modifying existing RSI patches causes undefined label selector errors

Applies to: 4.8.0, 4.8.1, and 4.8.2

Fixed in: 4.8.3

When you upgrade from IBM Cloud Pak for Data Version 4.6 or Version 4.7 to Version 4.8, you get an error when you modify existing RSI patches.

For example, if you want to make an active patch inactive, you encounter the following error when you run the cpd-cli manage create-rsi-patch command:

fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. 
The error was: 'dict object' has no attribute 'exclude_labels'...

Resolving the issue

To resolve the problem:

1. Run the following command to get the complete name of the patch:

   oc get zenextension --namespace=${PROJECT_CPD_INST_OPERANDS}

   RSI patches are prefixed with rsi-.

2. Set the RSI_PATCH_NAME environment variable to the name of the patch that you want to update:

   export RSI_PATCH_NAME=<patch-name>

3. Edit the patch:

   oc edit zenextension ${RSI_PATCH_NAME} \
   --namespace=${PROJECT_CPD_INST_OPERANDS}

4. Add the following entries to the spec.extensions[*].pod_selector section:

   "exclude_labels":{},
   "select_all_pods":false

   For example:

   apiVersion: zen.cpd.ibm.com/v1
   kind: ZenExtension
   metadata:
     name: rsi-zen-core-rsi-pod-env-var-json
   spec:
     extensions: |
       [
         {
           "extension_point_id": "rsi_pod_env_var",
           "extension_name": "rsi-zen-core-rsi-pod-env-var-json",
           "display_name": "rsi-zen-core-rsi-pod-env-var-json",
           "description": "description",
           "meta": {},
           "details": {
             "patch_spec": [
               {"op":"add","path":"/spec/containers/0/env/-","value":{"name":"zen-core-env-json-one","value":"zen-core-env-json-one"}},
               {"op":"add","path":"/spec/containers/0/env/-","value":{"name":"zen-core-env-json-two","value":"zen-core-env-json-two"}}
             ], 
            "pod_selector":{
               "selector":{
                  "app.kubernetes.io/component":"zen-core",
                  "component":"zen-core"
               },
               "exclude_labels":{}, 
               "select_all_pods":false
            },
             "state": "active",
             "type": "json"
           }
         }
       ]

5. Save your changes to the patch. For example, if you are using vi, hit esc and enter :wq.

Cannot get the contents of the zen-metastore-edb pod as a YAML file

Applies to: 4.8.0, 4.8.1, 4.8.2, and 4.8.3

Fixed in: 4.8.4

When you run the following command against any of the zen-metastore-edb pods, such as zen-metastore-edb-0, you get an error:

oc get pods zen-metastore-edb-0 -o yaml

The command returns the following error:

error: error converting JSON to YAML: yaml: control characters are not allowed

If you want to see the contents of the pod, you must use JSON. For example:

oc get pods zen-metastore-edb-0 -o json

The apply-cr command fails when installing services with a dependency on Db2U

Applies to: 4.8.0 and later

Diagnosing the problem

You can specify the privileges that Db2U runs with. If you configured Db2U to run with limited privileges, the apply-cr command will fail if:

• You set DB2U_RUN_WITH_LIMITED_PRIVS: "true" in the db2u-product-cm ConfigMap.
• The kernel parameter settings were not modified to allow Db2U to run with limited privileges.

This issue can manifest in several ways.

The wkc-db2u-init job fails

If you are installing IBM Knowledge Catalog, the apply-cr command fails with the message:

"WKC DB2U post install job failed ('wkc-db2u-init' job)

When you get the status of the wkc-db2u-init pods, they are in the Error state.

oc get pods -n ${PROJECT_CPD_INST_OPERANDS} | grep wkc-db2u-init

The Db2UCluster resource never becomes ready

For other services you might notice that the Db2uCluster resource never becomes Ready.

oc get Db2uCluster -n ${PROJECT_CPD_INST_OPERANDS}

You cannot provision service instances

For services such as Db2® and Db2 Warehouse, the apply-cr command completes successfully, but the service instances never finish provisioning and the *-db2u-0 pods are stuck in Pending or SysctlForbidden.

oc get pods -n ${PROJECT_CPD_INST_OPERANDS} | grep db2u-0

Resolving the problem

This problem occurs when you set DB2U_RUN_WITH_LIMITED_PRIVS: "true" in the db2u-product-cm ConfigMap but the kernel parameter settings were not modified to allow Db2U to run with limited privileges.

Review Changing kernel parameter settings to confirm that you can change the kernel parameter settings.

• If you can change the kernel parameter settings, ensure that the worker nodes are restarted after you change the settings.

  In some cases, when you run the cpd-cli manage apply-db2-kubelet command, the worker nodes are not restarted.

• If you cannot or do not change the kernel parameter settings, update the db2u-product-cm ConfigMap to set DB2U_RUN_WITH_LIMITED_PRIVS: "false". For more information, see Specifying the privileges that Db2U runs with.

Some Cloud Pak for Data software cannot be installed when the OpenShift Virtualization Operator is installed on the cluster

Applies to: 4.8.1, 4.8.2, 4.8.3 and 4.8.4

Fixed in: 4.8.5

Installing Red Hat OpenShift Container Platform for IBM Cloud Pak for Data previously stated that you could not install the OpenShift Virtualization Operator on the cluster. However, that restriction applies only to the environments with OpenPages or watsonx.governance Risk and Compliance Foundation.

After you upgrade from Cloud Pak for Data 4.7.4, generating a bearer token fails in an IAM-enabled cluster

Applies to: Upgrades from Version 4.7.4 to 4.8.0 and later

If you upgrade from Cloud Pak for Data version 4.7.4 to Cloud Pak for Data 4.8.0 or later, the IAM access token API (/idprovider/v1/auth/identitytoken) fails. You cannot log in to the user interface when the identitytoken API fails.

Diagnosing the problem

The following error is displayed in the log when you generate an IAM access token:

Failed to get access token, Liberty error: {"error_description":"CWWKS1406E: The token request had an invalid client credential. The request URI was \/oidc\/endpoint\/OP\/token.","error":"invalid_client"}"

Resolving the problem

1. Log in to Red Hat OpenShift Container Platform as a cluster administrator.

   ${OC_LOGIN}

   Remember: OC_LOGIN is an alias for the oc login command.
2. Run the following command to restart the oidc-client-registration job:

   oc -n delete job oidc-client-registration

After an upgrade from Cloud Pak for Data 4.7.3, FoundationDB can indicate a Failed status

Applies to: Upgrades from Version 4.7.3 to 4.8.2 and later

After upgrading Cloud Pak for Data from Version 4.7.3 to 4.8.2 or later, the status of the FoundationDB cluster can indicate that it has failed (fdbStatus: Failed). The Failed status can occur even if FoundationDB is available and working correctly. This issue occurs when the FoundationDB resources do not get properly cleaned up by the upgrade.

Diagnosing the problem

To determine if this problem has occurred:

Required role: To complete this task, you must be a cluster administrator.

1. Check the FoundationDB cluster status.

   oc get fdbcluster -o yaml | grep fdbStatus

   If the returned status is Failed, proceed to the next step to determine if the pods are available.

2. Check to see if the FoundationDB pods are up and running.

   oc get pods -n ${PROJECT_CPD_INST_OPERANDS} | grep foundation

   The returned list of FoundationDB pods should all have a status of Running. If they are not running, then the problem is something other than this issue.

Resolving the problem

To resolve this issue, restart the FoundationDB controller (ibm-fdb-controller):

Required role: To complete this task, you must be a cluster administrator.

1. Identify your FoundationDB controllers.

   oc get pods  -n ${PROJECT_CPD_INST_OPERATORS} | grep ibm-fdb-controller

   This command returns the names of two FoundationDB controllers in the following formats:
   • ibm-fdb-controller-manager-<INSTANCE-ID>
   • apple-fdb-controller-manager-<INSTANCE-ID>
2. Delete the ibm-fdb-controller-manager to refresh it.

   oc delete pod ibm-fdb-controller-<INSTANCE-ID> -n ${PROJECT_CPD_INST_OPERATORS}

3. Wait for the controller to restart. This can take approximately one minute.
4. Check the status of your FoundationDB cluster:

   oc -n ${PROJECT_CPD_INST_OPERANDS} get FdbCluster -o yaml

   Confirm that the fdbStatus is now Completed.

The apply-cr command fails when upgrading services with a dependency on Db2U

Applies to: 4.8.0, 4.8.1, 4.8.2, and 4.8.3

Fixed in: 4.8.4

When you try to upgrade services with a dependency on Db2U, the upgrade fails when trying to remove temporary files.

Diagnosing the problem

To determine if the upgrade failed when trying to remove temporary files:

1. Set the DB2U_POD environment variable to the name of the db2u-0 pod.
   • For IBM Knowledge Catalog, run:

     export DB2U_POD=$(oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep wkc-db2u-0)

   • For OpenPages:
     1. Set the INSTANCE_ID environment variable:

        export INSTANCE_ID=$(oc get openpagesinstance -n=${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.items[0].spec.zenServiceInstanceId}{"\n"}')

     2. Set the DB2U_POD environment variable:

        export DB2U_POD=$(oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep ${INSTANCE_ID}-db2u-0 | awk '{print $1}')

2. Open a remote shell in the pod to examine the upgrade_update.log file:

   oc exec -it ${DB2U_POD} \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -- bash -c "cat /mnt/blumeta0/support/upgrade_update.log"

3. Look for the following strings in the error log:

   • INFO: Applying Db2 license

   • in _delete_temporary_db2inst1_files

   • OSError: [Errno 16] Device or resource busy: '.nfs

   If the log contains the preceding strings, continue to Resolving the problem

Resolving the problem

1. Remove the db2.tmp_db2inst1.* files from the db2u-0 pod:

   oc exec -it ${DB2U_POD} \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -- bash -c "rm -rf db2.tmp_db2inst1.*"

2. Restart the database update:

   oc exec -it ${DB2U_POD} \
   -n=${PROJECT_CPD_INST_OPERANDS} \
   -- bash -c "db2_update_upgrade --databases"

3. Patch the formation:
   • For IBM Knowledge Catalog, run:

     oc patch formation db2oltp-wkc \
     -n=${PROJECT_CPD_INST_OPERANDS} \
     --type=json \
     --patch='[{"op": "remove", "path": "/spec/resource_configs/m/env/upgrade~1podname"}]'

   • For OpenPages, run:

     oc patch formation db2oltp-${INSTANCE_ID} \
     -n=${PROJECT_CPD_INST_OPERANDS} \
     --type=json \
     --patch='[{"op": "remove", "path": "/spec/resource_configs/m/env/upgrade~1podname"}]'

The apply-cr command fails when upgrading services with a dependency on the common core services

When you upgrade a service with a dependency on the common core services, the apply-cr command fails because the ccs-cr custom resource is stuck in InProgress. The problem can occur when the connection pods try to use an outdated certificate for inter-pod communication, which prevents the connection migration jobs from completing.

Diagnosing the problem

1. Get the status of the ccs-cr custom resource:

   oc get ccs -n=${PROJECT_CPD_INST_OPERANDS}

2. If the status of the custom resource is InProgress, check the status of the common core services pods on in the operands project for the instance (${PROJECT_CPD_INST_OPERANDS}):

   oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep ccs

   If the status of the ccs-post-install-migration-job pods are Error, contact IBM Software Support to restart the connection migration job.

The Projects page does not load after upgrade

Applies to: Upgrades from Version 4.8.1 to Version 4.8.2

After you upgrade from Version 4.8.1 to Version 4.8.2, the Projects page does not load.

Resolving the problem

To enable the Projects page to load:

1. Set the PORTAL_MAIN_POD to the name of the portal-main pod:

   export PORTAL_MAIN_POD=$(oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep portal-main | awk '{print $1}')

2. Delete the portal-main pod:

   oc delete pods ${PORTAL_MAIN_POD} -n=${PROJECT_CPD_INST_OPERANDS}

3. Wait for the pod to be Ready and try to load the Projects page again. To check the status of the pod, run:

   oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep portal-main

Unable to filter instances by type after you upgrade to Cloud Pak for Data 4.8.1

Applies to: 4.8.0 and 4.8.1

Fixed in: 4.8.2

Diagnosing the problem

After you upgrade to Cloud Pak for Data 4.8.1, you cannot filter by type on the Instances page. You see a blank page in the user interface.

Workaround

If you click Filter by: and then click Type, the page will become blank. You can refresh the page to return to the Instance page.

You must scroll through the Instances page to find a specific instance and its instance type.

After you upgrade Red Hat OpenShift Container Platform, storage volume pods are in the CrashLoopBackOff state

Applies to: 4.8.6 and later

After you upgrade Red Hat OpenShift Container Platform, storage volume pods (volumes-* pods are in the CrashLoopBackOff state. This problem occurs when the /auth/jwtpublic URL path in ibm-nginx returns a TLS error.

Diagnosing the problem

To confirm that the problem is caused by a TLS error returned by the /auth/jwtpublic URL path:

1. Get the name of a volumes-* pods that is in the CrashLoopBackOff state:

   oc get pods \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   | grep volumes

2. Get the pod logs for a volumes-* pod that is in the CrashLoopBackOff state:

   oc logs <pod-name> \
   --namespace=${PROJECT_CPD_INST_OPERANDS}

   Replace <pod-name> with the name of a pod in the CrashLoopBackOff state

3. Confirm that the logs include the following error:

   time="<timestamp>" level=error msg="ProcessPublicKeyFromNginx - Failed receiving response from server" func=zen-core-api/source/apis/commonutils.LogErr file="/go/src/zen-core-api/source/apis/commonutils/auth_util.go:342" err="Get \"https://ibm-nginx-svc.<namespace>:443/auth/jwtpublic\": remote error: tls: illegal parameter"
   panic: Get "https://ibm-nginx-svc.<namespace>:443/auth/jwtpublic": remote error: tls: illegal parameter

Resolving the problem

To resolve the problem:

1. Get the name of the volumes-* deployments:

   oc get deployments \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   | grep -E 'volumes|READY'

2. For each deployment where the READY column shows 0/1, run the following command to patch the deployment:

   oc patch deployment <volume-deployment-name> \
   --namespace=${PROJECT_CPD_INST_OPERANDS} \
   --type=json \
   --patch='[
          {"op": "add", "path": "/spec/template/spec/containers/0/volumeMounts/-", "value": {"mountPath": "/user-home/_global_/config/jwt", "name": "ibm-zen-secret-jwt"}},
          {"op": "add", "path": "/spec/template/spec/volumes/-", "value": {"name": "ibm-zen-secret-jwt", "secret": {"defaultMode": 420, "optional": true, "secretName": "ibm-zen-secret-jwt"}}}
        ]'

   Replace <volume-deployment-name> with the name of a deployment.

After you upgrade a Red Hat OpenShift Container Platform cluster, the FoundationDB resource can become unavailable

Applies to: 4.8.0 and later

After you upgrade your cluster to a new version of Red Hat OpenShift Container Platform, the IBM FoundationDB pods can become unavailable. When this issue occurs, services that rely on FoundationDB such as IBM Knowledge Catalog and IBM Match 360 cannot function correctly.

Diagnosing the problem

To identify the cause of this issue, check the FoundationDB status and details.

1. Check the FoundationDB status.

   oc get fdbcluster -o yaml | grep fdbStatus

   If this command is successful, the returned status is Complete. If the status is InProgress or Failed, proceed to the workaround steps.

2. If the status is Complete but FoundationDB is still unavailable, log in to one of the FDB pods and check the status details to ensure that the database is available and all coordinators are reachable.

   oc rsh sample-cluster-log-1 /bin/fdbcli

   To check the detailed status of the FDB pod, run fdbcli to enter the FoundationDB command-line interface, then run the following command at the fdb> prompt.

   status details

   • If you get a message that is similar to Could not communicate with a quorum of coordination servers, run the coordinators command with the IP addresses specified in the error message as input.

     oc get pod -o wide | grep storage
     > coordinators IP-ADDRESS-1:4500:tls IP-ADDRESS-2:4500:tls IP-ADDRESS-3:4500:tls 

     If this step does not resolve the problem, proceed to the workaround steps.

   • If you get a different message, such as Recruiting new transaction servers, proceed to the workaround steps.

Resolving the problem

To resolve this issue, restart the FoundationDB pods.

Required role: To complete this task, you must be a cluster administrator.

1. Restart the FoundationDB cluster pods.

   oc get fdbcluster 
   oc get po |grep ${CLUSTER_NAME} |grep -v backup|awk '{print }' |xargs oc delete po

   Replace ${CLUSTER_NAME} in the command with the name of your fdbcluster instance.

2. Restart the FoundationDB operator pods.

   oc get po |grep fdb-controller |awk '{print }' |xargs oc delete po

3. After the pods finish restarting, check to ensure that FoundationDB is available.
   1. Check the FoundationDB status.

      oc get fdbcluster -o yaml | grep fdbStatus

      The returned status must be Complete.

   2. Check to ensure that the database is available.

      oc rsh sample-cluster-log-1 /bin/fdbcli

      If the database is still not available, complete the following steps.

      1. Log in to the ibm-fdb-controller pod.
      2. Run the fix-coordinator script.

         kubectl fdb fix-coordinator-ips -c ${CLUSTER_NAME} -n ${PROJECT_CPD_INST_OPERATORS}

         Replace ${CLUSTER_NAME} in the command with the name of your fdbcluster instance.

         Note: For more information about the fix-coordinator script, see the workaround steps from the resolved IBM Match 360 known issue item The FoundationDB cluster can become unavailable.

Inaccurate status message from the command line after you upgrade

Diagnosing the problem

If you run the cpd-cli service-instance upgrade command from the Cloud Pak for Data command-line interface, and then use the service-instance list command to check the status of each service, the provision status for the service is listed as UPGRADE_FAILED.

Cause of the problem

When you upgrade the service, only the cpd-cli manage apply-cr command is supported. You cannot use the cpd-cli service-instance upgrade command to upgrade the service. And after you upgrade the service with the apply-cr method, the change in version and status is not recognized by the service-instance command. However, the correct version is displayed from the Cloud Pak for Data web client.

Resolving the problem

No action is required. If you use the cpd-cli manage apply-cr method to upgrade the service as documented, the upgrade is successful and you can ignore the version and status information that is generated by the cpd-cli service-instance list command.

Secrets are not visible in connections after upgrade

Applies to: Version 4.8.0, 4.8.1, 4.8.2, 4.8.3, and 4.8.4

If you use secrets when you create connections, the secrets are not visible in the connection details after you upgrade Cloud Pak for Data. This issue occurs when your vault uses a private CA signed certificate.

Resolving the problem

To see the secrets in the user interface:

1. Change to the project where Cloud Pak for Data is installed:

   oc project ${PROJECT_CPD_INST_OPERANDS}

2. Set the following environment variables:

   oc set env deployment/zen-core-api VAULT_BRIDGE_TLS_RENEGOTIATE=true
   oc set env deployment/zen-core-api VAULT_BRIDGE_TOLERATE_SELF_SIGNED=true

After upgrading to Red Hat OpenShift Container Platform Version 4.14, the status of the Redis custom resource fluctuates

Applies to: 4.8.0, 4.8.1, 4.8.2, and 4.8.3

Fixed in: 4.8.4

After you upgrade from Red Hat OpenShift Container Platform Version 4.12 to Version 4.14, the status of the Redis custom resource fluctuates between InProgress and Completed.

However, if the Redis operator (ibm-redis-cp-operator) is running without errors, you can ignore the fluctuation in the custom resource status.

Diagnosing the problem

1. To determine whether the custom resource status is fluctuating run the following command:

   oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep redis

   Wait several minutes and run the command again to see if the status changes.

2. To determine whether the operator pods are healthy, run the following command:

   oc get pods -n=${PROJECT_CPD_INST_OPERATORS} | grep ibm-redis-cp-operator

Resolving the problem

If the status of the operator pods are Running or Completed, you can ignore the fluctuation in the custom resource status.

If the status of the operator pods are not Running or Completed, follow the guidance in Troubleshooting the apply-olm command during installation or upgrade to determine the root cause of the problem.

Security scans return an Inadequate Account Lockout Mechanism message

Applies to: 4.8.0 and later

Diagnosing the problem

If you run a security scan against Cloud Pak for Data, the scan returns the following message.

Inadequate Account Lockout Mechanism

Resolving the problem

This is by design. It is strongly recommended that you use an enterprise-grade password management solution, such as SAML SSO or an LDAP provider for password management, as described in the following resources.

• Security on Cloud Pak for Data
• Managing Cloud Pak for Data users

The Kubernetes version information is disclosed

Applies to: 4.8.0 and later

Diagnosing the problem

If you run an Aqua Security scan against your cluster, the scan returns the following issue:

• KHV002 - Kubernetes issue disclosure

Resolving the problem

This is expected based on the following solution document from Red Hat OpenShift Customer Portal: Hide kubernetes /version API endpoint in OpenShift Container Platform 4.

Backup fails for the platform with error in EDB Postgres cluster

Applies to: 4.8.7 and later

Diagnosing the problem

For example, in IBM Storage Fusion, the backup fails at the Hook: br-service hooks/pre-backup stage in the backup sequence.

In the cpdbr-oadp.log file, you see the following error:

time=<timestamp> level=info msg=cmd stderr: Error: cannot take a cold backup of the primary instance 
or a target primary instance if the k8s.enterprisedb.io/snapshotAllowColdBackupOnPrimary annotation is not set to enabled

Cause of the problem

Labels and annotations in the EDB Postgres cluster resources were not updated after a switchover of the EDB Postgres cluster's primary instance and replica.

Resolving the problem

Use either the automatic or manual workaround.

Automatic workaround

After you apply the YAML files, the following workaround automatically runs as a prehook every time you take a backup. The issue is automatically handled, so you do not encounter it again, which is especially useful if you have set up automatic backups.

1. Check that the ${VERSION} environment variable is set in cpd_vars.sh to the correct Cloud Pak for Data version number.
2. Download the edb-patch-resources-legacy.yaml file.
3. Run the following command

   oc apply -n ${PROJECT_CPD_INST_OPERANDS} -f edb-patch-resources-legacy.yaml

4. Complete the steps that apply to your backup and restore method.
   Online backup and restore
   1. Download the edb-patch-aux-ckpt-cm-legacy.yaml file.
   2. Run the following command:

      sed "s/VERSION_PLACEHOLDER/${VERSION}/g" edb-patch-aux-ckpt-cm-legacy.yaml | oc apply -n ${PROJECT_CPD_INST_OPERANDS} -f -
      

   3. Retry the backup.

   ---

   Offline backup and restore
   1. Download the edb-patch-aux-br-cm-legacy.yaml file.
   2. Run the following command:

      sed "s/VERSION_PLACEHOLDER/${VERSION}/g" edb-patch-aux-br-cm-legacy.yaml | oc apply -n ${PROJECT_CPD_INST_OPERANDS} -f -
      

   3. Retry the backup.

   ---

Manual workaround

Complete the following steps to manually run the workaround:

Note: If another switchover of the EDB Postgres cluster's primary instance and replica happens after you apply the manual workaround, you must complete the workaround again before you take a backup.

1. Download the edb-patch.sh file.
2. Run the following command:

   sh edb-patch.sh ${PROJECT_CPD_INST_OPERANDS}

3. Retry the backup.

OADP backup is missing EDB Postgres PVCs

Applies to: 4.8.0 and later

Diagnosing the problem

After an online or offline backup is taken with the OADP utility, EDB Postgres PVCs are missing in the PVC backup list.

Cause of the problem

EDB Postgres replica PVCs might be excluded from a backup when an EDB Postgres cluster switches primary instances.

Resolving the problem

Before you create a backup, run the following command:

oc label pvc,pods -l k8s.enterprisedb.io/cluster,velero.io/exclude-from-backup=true velero.io/exclude-from-backup- -n ${PROJECT_CPD_INST_OPERANDS}

For more information, see the following topics:

• Creating an online backup with the Cloud Pak for Data OADP backup and restore utility
• Creating an offline backup of Cloud Pak for Data 4.8.0-4.8.4

OADP backup precheck command fails with EDB Postgres cluster is out of sync error

Applies to: 4.8.7

Fixed in: 4.8.8

Diagnosing the problem

When you run the cpd-cli oadp backup precheck command, you see the following error:

Error: precheck failed with error: edb in-sync check failed: edb cluster is out of sync

Cause of the problem

The backup precheck command is not accounting for any lag when log sequence numbers (LSNs) are received but not replayed yet.

Resolving the problem

Try one of the following options:

• Delete the EDB Postgres cluster's replica pod and PVC.

  For more information, see the troubleshooting topic PostgreSQL cluster replicas get out of sync.

• Wait for the LSNs to catch up.

Informix custom resource in InProgress state after restore

Applies to: 4.8.5

Fixed in: 4.8.8

Diagnosing the problem

After the restore, get the status of the Informix custom resource by running the following command:

oc get Informix informix-<xxxxxxxxxxxxxxxx>  -n ${PROJECT_CPD_INST_OPERANDS} -o yaml

In the output, the informixStatus shows InProgress.

Cause of the problem

The Informix custom resource stays in pending because the IBM Global Security Kit (GSKit) doesn't complete the SSL setup before the readiness and liveness checks complete.

Resolving the problem

Change the behavior of GSKit. From the OpenShift console, add the environment variable ICC_SHIFT=3 to the informix-<xxxxxxxxxxxxxxxx>-cm-0 and to the informix-<xxxxxxxxxxxxxxxx>-server StatefulSets, and restart those pods.

After restore, watsonx Assistant is stuck on the 17/19 deployed state or custom resource is stuck in InProgress state

Applies to: 4.8.5

Fixed in: 4.8.6

Diagnosing the problem

This problem can occur after you restore an online backup to the same cluster or to a different cluster. Run the following command:

oc get wa -n ${PROJECT_CPD_INST_OPERANDS}

Example output:

NAME   VERSION   READY   READYREASON    UPDATING   UPDATINGREASON   DEPLOYED   VERIFIED   QUIESCE   AGE
wa     5.0.1     False   Initializing   True       VerifyWait       17/19      15/19                4h39m

Resolving the problem

Delete the wa-integrations-operand-secret and wa-integrations-datastore-connection-strings secrets by running the following commands:

oc delete secret wa-integrations-operand-secret -n ${PROJECT_CPD_INST_OPERANDS}

oc delete secret wa-integrations-datastore-connection-strings -n ${PROJECT_CPD_INST_OPERANDS}

After the secrets are deleted, the watsonx Assistant operator recreates them with the correct values, and the watsonx Assistant custom resource and pods are now in a good state.

Missing Identity Management Service data after a restore

Applies to: 4.8.0-4.8.4

Fixed in: 4.8.5

Diagnosing the problem

After you upgrade to Cloud Pak for Data 4.8.0-4.8.4, you integrate with the Identity Management Service. When you back up and restore your Cloud Pak for Data deployment, Identity Management Service data is missing. No results are returned for one or more of the following commands:

oc get pvc -n ${PROJECT_CPD_INST_OPERANDS} ibm-zen-cs-mongo-backup

oc get cm -n ${PROJECT_CPD_INST_OPERANDS}  zen-cs-aux-br-cm

oc get cm -n ${PROJECT_CPD_INST_OPERANDS}  zen-cs-aux-ckpt-cm

Cause of the problem

The Cloud Pak for Data deployment is missing some resources, including several backup and restore ConfigMaps.

Resolving the problem

1. Copy and run the Identity Management Service backup and restore scripts.
2. Redo the backup and restore.

Unable to log in to Cloud Pak for Data with OpenShift cluster credentials after successfully restoring to a different cluster

Applies to: 4.8.5 and later

Diagnosing the problem

When Cloud Pak for Data is integrated with the Identity Management Service service, you cannot log in with OpenShift cluster credentials. You might be able to log in with LDAP or as cpdadmin.

Resolving the problem

To work around the problem, run the following commands:

oc delete cm platform-auth-idp -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm oauth-client-map -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm ibm-iam-bindinfo-oauth-client-map -n ${PROJECT_CPD_INST_OPERANDS}

oc delete cm ibm-iam-bindinfo-platform-auth-idp -n ${PROJECT_CPD_INST_OPERANDS}

oc delete pods -n ${PROJECT_CPD_INST_OPERATORS} -l app.kubernetes.io/instance=ibm-common-service-operator

oc delete pods -n ${PROJECT_CPD_INST_OPERANDS} -l app.kubernetes.io/instance=platform-auth-service

oc delete pods -n ${PROJECT_CPD_INST_OPERANDS} -l app.kubernetes.io/instance=platform-identity-management

oc delete pods -n ${PROJECT_CPD_INST_OPERANDS} -l app.kubernetes.io/instance=platform-identity-provider

Offline and online restore with OADP backup and restore utility fails with running post-restore hooks error

Applies to: 4.8.5-4.8.7

Fixed in: 4.8.8

Diagnosing the problem

The final cpd-cli oadp restore create command that runs post-restore hooks fails. In the CPD-CLI*.log file, you see the following error message:

error running post-restore hooks: pod/<podname> is not supported for scaling, please 
define the proper postrestore hooks because of restored replicaset(s)

Cause

A recent change in OADP 1.3.1 removed logic that prevents restoring deployment-managed ReplicaSets that shouldn't be restored.

Workaround

Do the following steps:

1. Find ReplicaSets that were restored from the backup:

   oc get rs -l velero.io/backup-name -n ${PROJECT_CPD_INST_OPERANDS}

2. Delete the ReplicaSets that are associated with the pods that have is not supported for scaling errors from the logs:

   oc delete rs -n ${PROJECT_CPD_INST_OPERANDS} <replicaset-name>

3. Manually run post-restore hooks.
   • If you are doing an offline backup and restore, run the following command:

     cpd-cli oadp restore posthooks --include-namespaces=${PROJECT_CPD_INST_OPERANDS},${PROJECT_CPD_INST_OPERATORS}

   • If you are doing an online backup and restore, run the following command:

     cpd-cli oadp restore posthooks --include-namespaces=${PROJECT_CPD_INST_OPERANDS},${PROJECT_CPD_INST_OPERATORS} --hook-kind=checkpoint

Restoring online backup of Watson Query fails

Applies to: 4.8.5 and later

Diagnosing the problem

In the CPD-CLI*.log file, you see the following error message:

time=<timestamp> level=info msg=   zen/configmap/cpd-dv-aux-ckpt-cm: component=dv, 
op=<mode=post-restore,type=config-hook,method=rule>, status=error func=cpdbr-oadp/pkg/quiesce.logPlanResult 
file=/a/workspace/oadp-upload/pkg/quiesce/planexecutor.go:1137

Workaround

Do the following steps:

1. Disable the Watson Query liveness probe in the Watson Query head pod:

   oc exec -it c-db2u-dv-db2u-0 -- bash -c "su - db2inst1 - mkdir /mnt/PV/versioned/marker_file"

   oc exec -it c-db2u-dv-db2u-0 -- bash -c "su - db2inst1 - touch /mnt/PV/versioned/marker_file/.bar"

2. Disable the BigSQL restart daemon in the Watson Query head pod:

   oc exec -it c-db2u-dv-db2u-0 -- bash -c "db2uctl marker create BIGSQL_DAEMON_PAUSE"

3. Stop BigSQL in the Watson Query head pod:

   oc rsh c-db2u-dv-db2u-0 bash

   su - db2inst1

   bigsql stop

4. Re-enable the Hive user in the users.json file in the Watson Query head pod.
   1. Edit the users.json file:

      vi /mnt/blumeta0/db2_config/users.json

   2. Locate "locked":true and change it to "locked":false.
5. On the hurricane pod, rename the hive-site.xml config file so that it can be reconfigured by restarting the pod:

   oc rsh $(oc get pod -o name -l formation_id=db2u-dv,role=hurricane)

   su - db2inst1

   mv /mnt/blumeta0/home/db2inst1/ibm/bigsql/hive-site.xml /mnt/blumeta0/home/db2inst1/ibm/bigsql/hive-site.xml.bak

6. Exit the pod, and then run the following command to delete it.
   Note: Since the configuration file was renamed, it is regenerated with the correct settings.

   oc delete pod -l formation_id=db2u-dv,role=hurricane

7. After the hurricane pod is started again, run the following commands on the hurricane pod to disable the SSL so that it can be reconfigured in a later step:

   oc rsh $(oc get pod -o name -l formation_id=db2u-dv,role=hurricane)

   su - db2inst1

   bigsql-config -disableMetastoreSSL

   bigsql-config -disableSchedulerSSL

8. Clean up leftover files from the hurricane pod:

   rm -rf /mnt/blumeta0/bigsql/security/*

   rm -rfv /mnt/blumeta0/bigsql/security/.* 2>/dev/null

9. Run the following commands to disable SSL from the head pod:

   oc rsh c-db2u-dv-db2u-0 bash

   su - db2inst1

   rah "bigsql-config -disableMetastoreSSL"

   rah "bigsql-config -disableSchedulerSSL"

10. Clean up leftover files from the head and worker pods:

    rm -rf /mnt/blumeta0/bigsql/security/*

    rm -rfv /mnt/blumeta0/bigsql/security/.* 2>/dev/null

11. Run the following commands to re-enable SSL on the head pod, and restart Db2 Big SQL so that configuration changes can take effect:

    bigsql-config -enableMetastoreSSL

    bigsql-config -enableSchedulerSSL

    bigsql stop; bigsql start

12. Remove markers that were created in steps 1 and 2 in the Watson Query head pod:

    oc exec -it c-db2u-dv-db2u-0 -- bash -c "rm -rf /mnt/PV/versioned/marker_file/.bar"

    oc exec -it c-db2u-dv-db2u-0 -- bash -c "db2uctl marker delete BIGSQL_DAEMON_PAUSE"

13. If you are doing the backup and restore with the OADP backup and restore utility, run the following command:

    cpd-cli oadp restore prehooks --hook-kind=checkpoint --include-namespaces=${PROJECT_CPD_INST_OPERANDS},${PROJECT_CPD_INST_OPERATORS} --log-level debug --verbose

14. If you are doing the backup and restore with IBM Storage Fusion, NetApp Astra Control Center, or Portworx data replication, run the following commands:

    CPDBR_POD=$(oc get po -l component=cpdbr-tenant -n ${PROJECT_CPD_INST_OPERATORS} --no-headers | awk '{print $1}')

    oc exec -n ${PROJECT_CPD_INST_OPERATORS} ${CPDBR_POD} -it -- /bin/sh -c "./cpdbr-oadp restore posthooks --hook-kind=checkpoint --include-namespaces=${PROJECT_CPD_INST_OPERANDS},${PROJECT_CPD_INST_OPERATORS}"

    oc exec -n ${PROJECT_CPD_INST_OPERATORS} ${CPDBR_POD} -it -- /bin/sh -c "./cpdops/files/cpd-operators.sh restore-namespacescope --foundation-namespace ${PROJECT_CPD_INST_OPERATORS} --operators-namespace ${PROJECT_CPD_INST_OPERATORS}"

After a restore, OperandRequest timeout error in the ZenService custom resource

Applies to: 4.8.5 and later

Diagnosing the problem

Get the status of the ZenService YAML:

oc get zenservice lite-cr -n ${PROJECT_CPD_INST_OPERATORS} -o yaml

In the output, you see the following error:

...
zenMessage: '5.1.3/roles/0010-infra has failed with error: "OperandRequest" "zen-ca-operand-request":
      Timed out waiting on resource'
...

Check for failing operandrequests:

oc get operandrequests -A

For failing operandrequests, check their conditions for constraints not satisfiable messages:

oc describe -n ${PROJECT_CPD_INST_OPERATORS} <opreq-name>

Cause of the problem

Subscription wait operations timed out. The problematic subscriptions show an error similar to the following example:

'constraints not satisfiable: clusterserviceversion ibm-db2aaservice-cp4d-operator.v5.2.0
      exists and is not referenced by a subscription, @existing/cpd-operators//ibm-db2aaservice-cp4d-operator.v5.2.0
      and ibm-db2aaservice-cp4d-operator-catalog/cpd-operators/v5.2/ibm-db2aaservice-cp4d-operator.v5.2.0
      originate from package ibm-db2aaservice-cp4d-operator, subscription ibm-db2aaservice-cp4d-operator
      requires ibm-db2aaservice-cp4d-operator-catalog/cpd-operators/v5.2/ibm-db2aaservice-cp4d-operator.v5.2.0,
      subscription ibm-db2aaservice-cp4d-operator exists'

This problem is a known issue with Red Hat Operator Lifecycle Manager. For details, see Failed to install or upgrade operator with warning 'clusterserviceversion is not referenced by a subscription'.

Workaround

Do the following steps:

1. Delete the problematic clusterserviceversions and subscriptions, and restart the Operand Deployment Lifecycle Manager (ODLM) pod.

   For details, follow the steps in the troubleshooting document cloud-native-postgresql operator is installed with the certified-operators catalogsource.

2. Delete Cloud Pak for Data instance projects (namespaces).

   For details, see Preparing to restore Cloud Pak for Data.

3. Retry the restore.

After a restore, the ZenService component remains in InProgress state

Applies to: 4.8.0, 4.8.1, 4.8.2

Fixed in: 4.8.3

Diagnosing the problem

In the backup and restore log file, you see the following message:

msg: Timeout creating internal-tls-certificate, make sure that certmanager has been installed and work properly

Workaround

Delete all restored certificate requests.

1. Get the list of all certificate requests:

   oc get certificaterequest -n ${PROJECT_CPD_INST_OPERANDS}

2. Delete each certificate request one-by-one:

   oc delete certificaterequest -n ${PROJECT_CPD_INST_OPERANDS} <cert-request-name>

Password for the cpadmin user changes when restoring Cloud Pak for Data

Applies to: 4.8.0-4.8.4

Fixed in: 4.8.5

Diagnosing the problem

This problem occurs when Cloud Pak for Data is integrated with the Identity Management Service service. The problem applies to all backup and restore methods.

After a restore, the password for the cpadmin user is changed.

Workaround

No workaround is available. To retrieve or change the new password, see Changing the administrator password.

After restoring IBM Match 360 from backup, the associated Redis pods enter a CrashLoopBackOff state

Fixed in: 4.8.3

Applies to: 4.8.1, 4.8.2

Diagnosing the problem

This issue occurs after restoring the IBM Match 360 from either an online or offline backup. The MDM Redis CP (mdm-redis-cp) pods fail to restore correctly and enter a CrashLoopBackOff state, which affects the functionality of the IBM Match 360 user interface services. This problem is caused by missing password fields in relevant secrets.

Workaround

To resolve this issue, set the administrator password field for RedisCP-related secrets:

1. Encode the user-defined password for Redis.

   echo -n '<USER-DEFINED-PASSWORD>' | base64

2. Run the following commands to set the administrator password to the encoded password value that you just created:

   oc -n ${PROJECT_CPD_INST_OPERANDS} patch secret  mdm-redis-cp-<INSTANCE-ID>-admin-secret --type=merge -p '{"data":{"admin_password": "<ENCODED-PASSWORD>"}}'

   oc -n ${PROJECT_CPD_INST_OPERANDS} patch secret  mdm-redis-cp-<INSTANCE-ID>-em-ui --type=merge -p '{"data":{"auth": "<ENCODED-PASSWORD>"}}'

3. Bring down the MDM RedisCP Server StatefulSet.

   oc patch sts mdm-redis-cp-<INSTANCE-ID>-server --type=merge -p '{"spec":{"replicas": 0}}'

   Wait for all MDM RedisCP pods to terminate.
4. Bring up the MDM RedisCP Server StatefulSet.

   oc patch sts mdm-redis-cp-<INSTANCE-ID>-server --type=merge -p '{"spec":{"replicas": 3}}'

5. Delete the MDM Redis HAProxy pod.

   oc delete pod -l redis-app=mdm-redis-cp-<INSTANCE-ID>-haproxy

Online backup of Watson Discovery fails at checkpoint stage

Applies to: 4.8.3

Fixed in: 4.8.4

Diagnosing the problem

When you try to create an online backup, the backup process fails at the checkpoint hook stage. For example, if you are creating the backup with IBM Storage Fusion, the backup process fails at the Hook: br-service-hooks-checkpoint stage in the backup sequence. In the log file, you see an error message similar to the following example:

download failed: s3://common-zen-wd/mt/__built-in-tenant__/fileResource/701db916-fc83-57ab-0000-000000000010.zip 
to tmp/s3-backup/common-zen-wd/mt/__built-in-tenant__/fileResource/701db916-fc83-57ab-0000-000000000010.zip 
Connection was closed before we received a valid response from endpoint URL: "https://s3.openshift-storage.svc:443/common-zen-wd/mt/__built-in-tenant__/fileResource/701db916-fc83-57ab-0000-000000000010.zip".

Cause of the problem

Large resource files can become corrupted while they are downloaded to the backup. As a result, the wd-discovery-aux-ch-s3-backup job does not complete successfully.

Workaround

Delete the file that is shown in the error message and recreate it.

1. Exec into the wd-discovery-support pod:

   oc exec -it deploy/wd-discovery-support -- bash

2. Do the following steps within the pod.
   1. Delete the file:

      aws-wd s3 rm s3://common-zen-wd/mt/__built-in tenant__/fileResource/<file_name>

   2. Confirm that the file is not listed when you run the following command:

      aws-wd s3 ls s3://common-zen-wd/mt/__built-in-tenant__/fileResource/

   3. Exit from the pod.
3. Delete the wd-discovery-orchestrator-setup job:

   oc delete job/wd-discovery-orchestrator-setup

4. Wait for the wd-discovery-orchestrator-setup job to run again and complete.

Confirm that the file was successfully recreated:

1. Exec into the wd-discovery-support pod:

   oc exec -it deploy/wd-discovery-support -- bash

2. Do the following steps within the pod.
   1. Copy the file to the tmp directory:

      aws-wd s3 cp s3://common-zen-wd/mt/__built-in-tenant__/fileResource/<file_name> /tmp

   2. Confirm that the file is copied:

      ls /tmp/<file_name>

   3. Exit from the pod.

You can now retake the backup, and the wd-discovery-aux-ch-s3-backup job will complete successfully.

Resource file causes online backup of Watson Discovery to fail at checkpoint stage

Applies to: 4.8.4, 4.8.5

Fixed in: 4.8.6

Diagnosing the problem

When you try to create an online backup, the backup process fails at the checkpoint hook stage. For example, if you are creating the backup with IBM Storage Fusion, the backup process fails at the Hook: br-service-hooks-checkpoint stage in the backup sequence. In the log file, you see an error message similar to the following example:

download failed: s3://common-zen-wd/user/1000750000/.sparkStaging/application_1710244391677_0005/training-job.jar 
to tmp/s3-backup/common-zen-wd/user/1000750000/.sparkStaging/application_1710244391677_0005/training-job.jar 
Connection was closed before we received a valid response from endpoint 
URL: "https://s3.openshift-storage.svc:443/common-zen-wd/user/1000750000/.sparkStaging/application_1710244391677_0005/training-job.jar".

Cause of the problem

Resource files can become corrupted, and as a result, the wd-discovery-aux-ch-s3-backup job does not complete successfully.

Workaround

Delete the file that is shown in the error message from s3 and re-run the backup.

1. Exec into the wd-discovery-support pod:

   oc exec -it deploy/wd-discovery-support -- bash

2. Do the following steps within the pod.
   1. Delete the file:

      aws-wd s3 rm <file_name>

      For example:

      aws-wd s3 rm s3://common-zen-wd/user/1000750000/.sparkStaging/application_1710244391677_0005/training-job.jar`

   2. Confirm that the file was deleted:

      aws-wd s3 ls <file_name>

   3. Exit from the pod.

You can now retake the backup, and the wd-discovery-aux-ch-s3-backup job will complete successfully.

Watson Knowledge Catalog custom resource stuck in InProgress state after restore

Applies to: 4.8.9

Diagnosing the problem

When you try to restore, the Watson Knowledge Catalog custom resource is stuck in an InProgress state, and you see a message similar to the following example:

NAME     VERSION   RECONCILED   STATUS       AGE
wkc-cr   4.8.9                  InProgress   5h36m


  lastTransitionTime: "2025-04-16T17:08:20Z"
      message: |-
        unknown playbook failure
        Failed to deploy WKC prereqs
        Failed at task: Wait until FdbCluster is completed
        The error was: Please consult the operator logs.

Cause of the problem

Corrupted or incomplete snapshot data is causing unrecoverable indexes in OpenSearch during the restore process.

Workaround

Complete the following steps to resolve the issue:

1. Reduce GS replicas to 0:

   oc scale deployment wkc-search --replicas=0

2. Change the readiness check configuration to accept red as a valid status:

   oc patch ccs ccs-cr --type merge --patch '{"spec": {"openshift_cluster_health_check_params": "wait_for_status=red&timeout=30s"}}'

3. Put the OpenSearch cluster in quiesce mode:

   oc patch elasticsearchcluster elasticsearch-master --type merge --patch '{"spec": {"quiesce": true}}'

4. Take the OpenSearch cluster out of quiesce mode:

   oc patch elasticsearchcluster elasticsearch-master --type merge --patch '{"spec": {"quiesce": false}}'

5. Verify the presence of corrupted indexes, which are shards that are stuck and do not respond to recovery:

   oc exec elasticsea-0ac3-ib-6fb9-es-server-esnodes-0 -c elasticsearch -- curl --request GET --url http://localhost:19200/_cat/shards  --header 'content-type: application/json'

6. Delete the corrupted indexes:

   oc exec elasticsea-0ac3-ib-6fb9-es-server-esnodes-0 -c elasticsearch -- curl --request DELETE --url http://localhost:19200/gs-system-index-wkc-v001,semantic,wkc --header 'content-type: application/json'

7. Put the OpenSearch cluster in quiesce mode:

   oc patch elasticsearchcluster elasticsearch-master --type merge --patch '{"spec": {"quiesce": true}}'

8. Change the readiness check configuration back to the default setting, which accepts only yellow as a valid status:

   oc patch ccs ccs-cr --type merge --patch '{"spec": {"openshift_cluster_health_check_params": "wait_for_status=yellow&timeout=30s"}}'

9. Take the OpenSearch cluster out of quiesce mode:

   oc patch elasticsearchcluster elasticsearch-master --type merge --patch '{"spec": {"quiesce": false}}'

10. Scale the wkc-search pod back to its original size:

    oc scale deployment wkc-search --replicas=1

You can now complete the restore successfully.

The Flight service do_action command might show misleading error messages in Db2 table drop or create operations

Applies to: 4.8.0

Fixed in: 4.8.1

Diagnosing the problem

If you run the do_action command from Jupyter Notebooks with Python, you might see error messages that drop or create statements in a Db2 table were not successful. For example:

../src/arrow/status.cc:137: DoAction result was not fully consumed: Cancelled: Flight cancelled call, with message: Cancelled. Detail: Cancelled

Workaround

The behavior of do_action has changed. You must iterate through the results of do_action to ensure that all results are collected. For example,

results = list(flight_client.do_action(action))
return [r.body.to_pybytes().decode('utf-8') for r in results]