Known issues and limitations for IBM Software Hub

The following issues apply to the IBM® Software Hub platform. Each issue includes information about the releases that it applies to. If the issue was fixed in a refresh, that information is also included.

The following issues apply to IBM Software Hub services.

Customer-reported issues

Issues that are found after the release are posted on the IBM Support site.

General issues

After rebooting a cluster that uses OpenShift Data Foundation storage, some IBM Software Hub services aren't functional
Auxiliary modules are not found when running export-import commands on ppc64le hardware
Export jobs are not deleted if the export specification file includes incorrectly formatted JSON
The health service-functionality check fails at certain API endpoints for AI Factsheets, Data Product Hub, OpenPages, and watsonx.data
The health service-functionality check fails on the v2/ingestion_jobs API endpoint for watsonx.data
Cannot scale down IBM Knowledge Catalog

After rebooting a cluster that uses OpenShift Data Foundation storage, some IBM Software Hub services aren't functional

Applies to: 5.2.0 and later

Diagnosing the problem

After rebooting the cluster, some IBM Software Hub custom resources remain in the InProgress state.

For more information about this problem, see Missing NodeStageVolume RPC call blocks new pods from going into Running state in the Red Hat® OpenShift® Data Foundation 4.1.4 release notes.

Workaround

To enable the pods to come up after a reboot:

Find the nodes that have pods that are in an Error state:

oc get po -n ${PROJECT_CPD_INST_OPERANDS} -o wide s -A  | grep -v -P "Completed|(\d+)\/\1"

Mark each node as unschedulable.
```
oc adm cordon <node_name>
```

Delete the affected pods:

oc get pod   | grep -Ev "Comp|0/0|1/1|2/2|3/3|4/4|5/5|6/6|7/7"|awk '{print $1}' |xargs oc delete po --force=true --grace-period=0

Mark each node as scheduled:
```
oc adm uncordon <node_name>
```

Auxiliary modules are not found when running `export-import` commands on ppc64le hardware

Applies to: 5.2.0

Fixed in: 5.2.1

When you run the cpd-cli export-import list aux-module command on ppc64le hardware, the command returns:

No module(s) found

Resolving the problem

To run the export-import commands on ppc64le hardware, you must edit the wkc-base-aux-exim-cm ConfigMap:

Print the contents of the wkc-base-aux-exim-cm ConfigMap to a file called wkc-base-aux-exim-cm.yaml:
```
oc get cm wkc-base-aux-exim-cm \
-n ${PROJECT_CPD_INST_OPERANDS} \
-o yaml > wkc-base-aux-exim-cm.yaml
```
Open the wkc-base-aux-exim-cm.yaml file in a text editor.
Locate the data.aux-meta section.

Update the architecture parameter to ppc64le:

data:
  aux-meta: |
    name: wkc-base-aux
    description: wkc-base auxiliary export/import module
    architecture: "ppc64le"
    ...

Locate the metadata.labels section.

Update the cpdfwk.arch parameter to ppc64le:

metadata:
  labels:
    app: wkc-base
    app.kubernetes.io/instance: wkc-base
    app.kubernetes.io/managed-by: Tiller
    app.kubernetes.io/name: wkc-base
    chart: wkc-base
    cpdfwk.arch: ppc64le
    ...

Save your changes to the wkc-base-aux-exim-cm.yaml file.
Apply your changes to the wkc-base-aux-exim-cm ConfigMap:
```
oc apply -f wkc-base-aux-exim-cm.yaml
```

Export jobs are not deleted if the export specification file includes incorrectly formatted JSON

Applies to: 5.2.0 and later

When you export data from IBM Software Hub, you must create an export specification file. The export specification file includes a JSON string that defines the data that you want to export.

If the JSON string is incorrectly formatted, the export will fail. In addition, if you try to run the cpd-cli export-import export delete command, the command completes without returning any errors, but the export job is not deleted.

Diagnosing the problem

To confirm that the export job was not deleted, run:

cpd-cli export-import export status \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--profile=${CPD_PROFILE_NAME}

Resolving the problem

Use the OpenShift Container Platform command-line interface to delete the failed export job:
1. Set the EXPORT_JOB environment variable to the name of the job that you want to delete:
```
export EXPORT_JOB=<export-job-name>
```
2. To remove the export job, run:
```
oc delete job ${EXPORT_JOB} \
--namespace=${PROJECT_CPD_INST_OPERANDS}
```
If you want to use the export specification file to export data, fix the JSON formatting issues.

The health service-functionality check fails at certain API endpoints for AI Factsheets, Data Product Hub, OpenPages, and watsonx.data

Applies to: 5.2.0

Fixed in: 5.2.1

The

cpd-cli
          health
          service-functionality

checks fail at specific API endpoints for the following services:

AI Factsheets

API endpoint: /model_inventory/model_entries/

This failure is not indicative of a problem with AI Factsheets. There is no impact on the functionality of the service if this check fails.

Data Product Hub

API endpoint: /retire

This failure is not indicative of a problem with Data Product Hub. There is no impact on the functionality of the service if this check fails.

OpenPages

API endpoint: /workflows/definitions/68/start/2511

This failure is not indicative of a problem with OpenPages. There is no impact on the functionality of the service if this check fails.

watsonx.data™

API endpoint: /lakehouse/api/v2/upload/json?engine_id=presto-01

This failure is not indicative of a problem with watsonx.data. There is no impact on the functionality of the service if this check fails.

The health service-functionality check fails on the `v2/ingestion_jobs` API endpoint for watsonx.data

Applies to: 5.2.2

API endpoint: /lakehouse/api/v2/ingestion_jobs

This failure is not indicative of a problem with watsonx.data. There is no impact on the functionality of the service if this check fails.

Cannot scale down IBM Knowledge Catalog

Applies to: 5.2.1

Fixed in: 5.2.2

If you scale up IBM Knowledge Catalog, for example to level_3(medium), it increases the number of Neo4j cluster primaries. However, you cannot scale the number of Neo4j cluster back down once increased. This limitation prevents data integrity issues.

In Software Hub 5.2.1, scaling down might appear to be successful. However, in this case, the Neo4jCluster is in a Failed state even if the three Neo4j servers are up and running. This situation can cause other issues. For example, after you shut down IBM Knowledge Catalog, it can fail to restart.

To fix the issue, you can set the number of clusterPrimaries to 3 by using the following command:

oc patch neo4jcluster datalineage-neo4j \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type=merge
--patch='{"spec":{"clusterPrimaries": 3}}'

Installation and upgrade issues

The `setup-instance` command fails waiting for the `ibm-namespace-scope-operator` operator to be installed or upgraded

Applies to: 5.2.1

Fixed in: 5.2.2

When you run the cpd-cli manage setup-instance command, the command fails while waiting for the ibm-namespace-scope-operator operator to be installed or upgraded.

The command fails with the following error:

/tmp/work/cpfs_scripts/5.2.1/cp3pt0-deployment/common/utils.sh at line 127 in function wait_for_condition: 
Timeout after 40 minutes waiting for operator ibm-namespace-scope-operator to be upgraded

This issue occurs when your cluster pulls images from a private container registry and one or more of the following images cannot be found:

ibm-cpd-cloud-native-postgresql-operator-bundle@sha256:525924934f74b28783dacf630a6401ccd4c88d253aafa2d312b7993d8b9a14c1
ibm-cpd-cloud-native-postgresql-operator@sha256:a72c2b70a9f0a93047d47247cddc1616138cbc87337289ddb208565af6e4369d

The `setup-instance` command fails if the `ibm-common-service-operator-service` service is not found

Applies to: Upgrades from:

4.8.x
5.0.x
5.1.x
5.2.x

When you run the cpd-cli manage setup-instance command, the command fails if the ibm-common-service-operator-service service is not found in the operator project for the instance.

When this error occurs, the ibmcpd ibmcpd-cr custom resource is stuck at 35%.

Diagnosing the problem

To determine if the command failed because the ibm-common-service-operator-service service was not found:

Get the .status.progress value from the ibmcpd ibmcpd-cr custom resource:
```
oc get ibmcpd ibmcpd-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
-o json | jq -r '.status.progress'
```
- If the command returns 35%, continue to the next step.
- If the command returns a different value, the command failed for a different reason.
Check for the Error found when checking commonservice CR in namespace error in the ibmcpd ibmcpd-cr custom resource:
```
oc get ibmcpd ibmcpd-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
-o json | grep 'Error found when checking commonservice CR in namespace'
```
- If the command returns a response, continue to the next step.
- If the command does not return a response, the command failed for a different.

Confirm that the ibm-common-service-operator-service service does not exist:

oc get svc ibm-common-service-operator-service \
--namespace=${PROJECT_CPD_INST_OPERATORS}

The command should return the following response:

Error from server (NotFound): services "ibm-common-service-operator-service" not found

Resolving the problem

To resolve the problem:

Get the name of the cpd-platform-operator-manager pod:

oc get pod \
--namespace=${PROJECT_CPD_INST_OPERATORS} \
| grep cpd-platform-operator-manager

Delete the cpd-platform-operator-manager pod.
Replace <pod-name> with the name of the pod returned in the previous step.
```
oc delete pod <pod-name> \
--namespace=${PROJECT_CPD_INST_OPERATORS}
```
Wait several minutes for the operator to run a reconcile loop.

Confirm that the ibm-common-service-operator-service service exists:

oc get svc ibm-common-service-operator-service \
--namespace=${PROJECT_CPD_INST_OPERATORS}

The command should return a response with the following format:

NAME                                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
ibm-common-service-operator-service   ClusterIP   198.51.100.255   <none>        443/TCP   1h

After you resolve the issue, you can re-run the cpd-cli manage setup-instance command.

The Switch locations icon is not available if the `apply-cr` command times out

Applies to: 5.2.0 and later

If you install solutions that are available in different experiences, the Switch locations icon ( Image of the Switch locations icon. ) is not available in the web client if the cpd-cli manage apply-cr command times out.

Resolving the problem: Re-run the cpd-cli manage apply-cr command.

Upgrades fail if the Data Foundation Rook Ceph cluster is unstable

Applies to: 5.2.0 and later

If the Red Hat OpenShift Data Foundation or IBM Fusion Data Foundation Rook Ceph® cluster is unstable, upgrades fail.

One symptom is that pods will not start because of a FailedMount error. For example:

Warning  FailedMount  36s (x1456 over 2d1h)   kubelet  MountVolume.MountDevice failed for volume 
"pvc-73bf3705-43e9-40bd-87ed-c1e1656d6f12" : rpc error: code = Aborted desc = an operation with the given 
Volume ID 0001-0011-openshift-storage-0000000000000001-5e17508b-c295-4306-b684-eaa327aec2ab already exists

Diagnosing the problem

To confirm whether the Data Foundation Rook Ceph cluster is unstable:

Ensure that the rook-ceph-tools pod is running.
```
oc get pods -n openshift-storage | grep rook-ceph-tools
```
Note: On IBM Fusion HCI System or on environments that use hosted control planes, the pods are running in the openshift-storage-client project.
Set the TOOLS_POD environment variable to the name of the rook-ceph-tools pod:
```
export TOOLS_POD=<pod-name>
```

Execute into the rook-ceph-tools pod:

oc rsh -n openshift-storage ${TOOLS_POD}

Run the following command to get the status of the Rook Ceph cluster:
```
ceph status
```
Confirm that the output includes the following line:
```
health: HEALTH_WARN
```
Exit the pod:
```
exit
```

Resolving the problem

To resolve the problem:

Get the name of the rook-ceph-mrg pods:

oc get pods -n openshift-storage | grep rook-ceph-mgr

Set the MGR_POD_A environment variable to the name of the rook-ceph-mgr-a pod:
```
export MGR_POD_A=<rook-ceph-mgr-a-pod-name>
```
Set the MGR_POD_B environment variable to the name of the rook-ceph-mgr-b pod:
```
export MGR_POD_B=<rook-ceph-mgr-b-pod-name>
```

Delete the rook-ceph-mgr-a pod:

oc delete pods ${MGR_POD_A} -n openshift-storage

Ensure that the rook-ceph-mgr-a pod is running before you move to the next step:
```
oc get pods -n openshift-storage | grep rook-ceph-mgr
```

Delete the rook-ceph-mgr-b pod:

oc delete pods ${MGR_POD_B} -n openshift-storage

Ensure that the rook-ceph-mgr-b pod is running:

oc get pods -n openshift-storage | grep rook-ceph-mgr

After you upgrade a Red Hat OpenShift Container Platform cluster, the FoundationDB resource can become unavailable

Applies to: 5.2.0 and later

After you upgrade your cluster to a new version of Red Hat OpenShift Container Platform, the IBM FoundationDB pods can become unavailable. When this issue occurs, services that rely on FoundationDB such as IBM Knowledge Catalog and IBM Match 360 cannot function correctly.

This issue affects deployments of the following services.

IBM Knowledge Catalog
IBM Match 360

Diagnosing the problem

To identify the cause of this issue, check the FoundationDB status and details.

Check the FoundationDB status.
```
oc get fdbcluster -o yaml | grep fdbStatus
```
If this command is successful, the returned status is Complete. If the status is InProgress or Failed, proceed to the workaround steps.
If the status is Complete but FoundationDB is still unavailable, log in to one of the FDB pods and check the status details to ensure that the database is available and all coordinators are reachable.
```
oc rsh sample-cluster-log-1 /bin/fdbcli
```
To check the detailed status of the FDB pod, run fdbcli to enter the FoundationDB command-line interface, then run the following command at the fdb> prompt.
```
status details
```
- If you get a message that is similar to Could not communicate with a quorum of coordination servers, run the coordinators command with the IP addresses specified in the error message as input.
```
oc get pod -o wide | grep storage
> coordinators IP-ADDRESS-1:4500:tls IP-ADDRESS-2:4500:tls IP-ADDRESS-3:4500:tls 
```
  If this step does not resolve the problem, proceed to the workaround steps.
- If you get a different message, such as Recruiting new transaction servers, proceed to the workaround steps.

Resolving the problem

To resolve this issue, restart the FoundationDB pods.

Required role: To complete this task, you must be a cluster administrator.

Restart the FoundationDB cluster pods.
```
oc get fdbcluster 
oc get po |grep ${CLUSTER_NAME} |grep -v backup|awk '{print }' |xargs oc delete po
```
Replace ${CLUSTER_NAME} in the command with the name of your fdbcluster instance.

Restart the FoundationDB operator pods.

oc get po |grep fdb-controller |awk '{print }' |xargs oc delete po

After the pods finish restarting, check to ensure that FoundationDB is available.
1. Check the FoundationDB status.
```
oc get fdbcluster -o yaml | grep fdbStatus
```
  The returned status must be Complete.
2. Check to ensure that the database is available.
```
oc rsh sample-cluster-log-1 /bin/fdbcli
```
  If the database is still not available, complete the following steps.
  1. Log in to the ibm-fdb-controller pod.
  2. Run the fix-coordinator script.
```
kubectl fdb fix-coordinator-ips -c ${CLUSTER_NAME} -n ${PROJECT_CPD_INST_OPERATORS}
```
    Replace ${CLUSTER_NAME} in the command with the name of your fdbcluster instance.

Persistent volume claims with the `WaitForFirstConsumer` volume binding mode are flagged by the installation health checks

Applies to: 5.2.0 and later

When you install IBM Software Hub, the following persistent volume claims are automatically created:

ibm-cs-postgres-backup
ibm-zen-objectstore-backup-pvc

Both of these persistent volume claims are created with the WaitForFirstConsumer volume binding mode. In addition, both persistent volume claims will remain in the Pending state until you back up your IBM Software Hub installation. This behavior is expected. However, when you run the cpd-cli health operands command, the Persistent Volume Claim Healthcheck fails.

If there are more persistent volume claims returned by the health check, you must investigate further to determine why those persistent volume claims are pending. However, if only the following persistent volume claims are returned, you can ignore the Failed result:

ibm-cs-postgres-backup
ibm-zen-objectstore-backup-pvc

Node pinning is not applied to `postgresql` pods

Applies to: 5.2.0 and later

If you use node pinning to schedule pods on specific nodes, and your environment includes postgresql pods, the node affinity settings are not applied to the postgresql pods that are associated with your IBM Software Hub deployment.

The resource specification injection (RSI) webhook cannot patch postgresql pods because the EDB Postgres operator uses a PodDisruptionBudget resource to limit the number of concurrent disruptions to postgresql pods. The PodDisruptionBudget resource prevents postgresql pods from being evicted.

The `ibm-nginx` deployment does not scale fast enough when automatic scaling is configured

Applies to: 5.2.0 and later

If you configure automatic scaling for IBM Software Hub, the ibm-nginx deployment might not scale fast enough. Some symptoms include:

Slow response times
High CPU requests are throttled
The deployment scales up and down even when the workload is steady

This problem typically occurs when you install watsonx Assistant or watsonx™ Orchestrate.

Resolving the problem

If you encounter the preceding symptoms, you must manually scale the ibm-nginx deployment:

oc patch zenservice lite-cr \
--namespace=${PROJECT_CPD_INST_OPERANDS} \
--type merge \
--patch '{"spec": {
    "Nginx": {
        "name": "ibm-nginx",
        "kind": "Deployment",
        "container": "ibm-nginx-container",
        "replicas": 5,
        "minReplicas": 2,
        "maxReplicas": 11,
        "guaranteedReplicas": 2,
        "metrics": [
            {
                "type": "Resource",
                "resource": {
                    "name": "cpu",
                    "target": {
                        "type": "Utilization",
                        "averageUtilization": 529
                    }
                }
            }
        ],
        "resources": {
            "limits": {
                "cpu": "1700m",
                "memory": "2048Mi",
                "ephemeral-storage": "500Mi"
            },
            "requests": {
                "cpu": "225m",
                "memory": "920Mi",
                "ephemeral-storage": "100Mi"
            }
        },
        "containerPolicies": [
            {
                "containerName": "*",
                "minAllowed": {
                    "cpu": "200m",
                    "memory": "256Mi"
                },
                "maxAllowed": {
                    "cpu": "2000m",
                    "memory": "2048Mi"
                },
                "controlledResources": [
                    "cpu",
                    "memory"
                ],
                "controlledValues": "RequestsAndLimits"
            }
        ]
    }
}}'

Uninstalling IBM watsonx services does not remove the IBM watsonx experience

Applies to: 5.2.0 and later

After you uninstall watsonx.ai™ or watsonx.governance™, the IBM watsonx experience is still available in the web client even though there are no services that are specific to the IBM watsonx experience.

Resolving the problem

To remove the IBM watsonx experience from the web client, an instance administrator must run the following command:

oc delete zenextension wx-perspective-configuration \
--namespace=${PROJECT_CPD_INST_OPERANDS}

Backup and restore issues

Issues that apply to several backup and restore methods

Backup issues

Review the following issues before you create a backup. Do the workarounds that apply to your environment.

Restore issues

Review the following issues before you restore a backup. Do the workarounds that apply to your environment.

Backup and restore issues with the OADP utility

Backup issues

Review the following issues before you create a backup. Do the workarounds that apply to your environment.

Restore issues

Review the following issues after you restore a backup. Do the workarounds that apply to your environment.

Backup and restore issues with IBM Fusion

Backup issues

Review the following issues before you create a backup. Do the workarounds that apply to your environment.

Restore issues

Do the workarounds that apply to your environment after you restore a backup.

Backup and restore issues with Portworx

Backup issues

Review the following issues after you restore a backup. Do the workarounds that apply to your environment.

IBM Software Hub resources are not migrated

ObjectBucketClaim is not supported by the OADP utility

Applies to: 5.2.0 and later

Applies to: Backup and restore with the OADP utility

Diagnosing the problem: If an ObjectBucketClaim is created in an IBM Software Hub instance, it is not included when you create a backup.
Cause of the problem: OADP does not support backup and restore of ObjectBucketClaim.
Resolving the problem: Services that provide the option to use ObjectBuckets must ensure that the ObjectBucketClaim is in a separate namespace and backed up separately.

After a restore, OperandRequest timeout error in the ZenService custom resource

Applies to: 5.2.0 and later

Applies to: All backup and restore methods

Diagnosing the problem

Get the status of the ZenService YAML:

oc get zenservice lite-cr -n ${PROJECT_CPD_INST_OPERATORS} -o yaml

In the output, you see the following error:

...
zenMessage: '5.1.3/roles/0010-infra has failed with error: "OperandRequest" "zen-ca-operand-request":
      Timed out waiting on resource'
...

Check for failing operandrequests:

oc get operandrequests -A

For failing operandrequests, check their conditions for

constraints not
                satisfiable

messages:

oc describe -n ${PROJECT_CPD_INST_OPERATORS} <opreq-name>

Cause of the problem

Subscription wait operations timed out. The problematic subscriptions show an error similar to the following example:

'constraints not satisfiable: clusterserviceversion ibm-db2aaservice-cp4d-operator.v5.2.0
      exists and is not referenced by a subscription, @existing/cpd-operators//ibm-db2aaservice-cp4d-operator.v5.2.0
      and ibm-db2aaservice-cp4d-operator-catalog/cpd-operators/v5.2/ibm-db2aaservice-cp4d-operator.v5.2.0
      originate from package ibm-db2aaservice-cp4d-operator, subscription ibm-db2aaservice-cp4d-operator
      requires ibm-db2aaservice-cp4d-operator-catalog/cpd-operators/v5.2/ibm-db2aaservice-cp4d-operator.v5.2.0,
      subscription ibm-db2aaservice-cp4d-operator exists'

This problem is a known issue with Red Hat Operator Lifecycle Manager. For details, see Failed to install or upgrade operator with warning 'clusterserviceversion is not referenced by a subscription'.

Resolving the problem

Do the following steps:

Delete the problematic clusterserviceversions and subscriptions, and restart the Operand Deployment Lifecycle Manager (ODLM) pod.
For details, follow the steps in the troubleshooting document cloud-native-postgresql operator is installed with the certified-operators catalogsource.
Delete IBM Software Hub instance projects (namespaces).
For details, see Cleaning up the cluster before a restore.
Retry the restore.

After backing up or restoring IBM Match 360, Redis fails to return to a `Completed` state

Fixed in: 5.2.1

Applies to: 5.2.0

Applies to: All backup and restore methods

Diagnosing the problem

After completing a backup or restoring the IBM Match 360 service, its associated Redis pod (mdm-redis) and IBM Redis CP fail to return to a Completed state.

Cause of the problem

This occurs when Redis CP fails to come out of maintenance mode.

Resolving the problem

To resolve this issue, manually update Redis CP to take it out of maintenance mode:

Open the Redis CP YAML file for editing.

oc edit rediscp -n ${PROJECT_CPD_INST_OPERANDS} -o yaml

Update the value of the ignoreForMaintenance parameter from true to false.
```
ignoreForMaintenance: false
```
Save the file and wait for the Redis pod to reconcile.

Watson OpenScale fails after restore due to Db2 `(db2oltp)` or Db2 Warehouse `(db2wh)` configuration

Applies to: 5.2.0 and later

Applies to: All backup and restore methods

Diagnosing the problem

After you restore, Watson OpenScale fails due to memory constraints. You might see Db2 (db2oltp) or Db2 Warehouse (db2wh) instances that return 404 errors and pod failures where scikit pods are unable to connect to Apache Kafka.

Cause of the problem

The root cause is typically insufficient memory or temporary table page size settings, which are critical for query execution and service stability.

Resolving the problem

Ensure that the Db2 (db2oltp) or Db2 Warehouse (db2wh) instance is configured with adequate memory resources, specifically:

Set the temporary table page size to at least 700 GB during instance setup or reconfiguration.
Monitor pod health and Apache Kafka connectivity to verify that dependent services recover after memory allocation is corrected.

Watson OpenScale fails after restore for some Watson Machine Learning deployments

Applies to: 5.2.0 and later

Applies to: All backup and restore methods

Diagnosing the problem

After restoring an online backup, automatic payload logging in Watson OpenScale fails for certain Watson Machine Learning deployments.

Cause of the problem

The failure is caused by a timing issue during the restore process. Specifically, the Watson OpenScale automatic setup fails because Watson Machine Learning runtime pods are unable to connect to the underlying Apache Kafka service. This connectivity issue occurs because the pods start before Apache Kafka is fully available.

Resolving the problem

To resolve the issue, restart all wml-dep pods after you restore. This ensures proper Apache Kafka connectivity and allows Watson OpenScale automatic setup to complete successfully:

List all wml-dep pods:

oc get pods -n=${PROJECT_CPD_INST_OPERANDS} | grep wml-dep

Run the following command for each wml-dep pod to delete it and trigger it to restart:
```
oc delete pod <podname> -n=${PROJECT_CPD_INST_OPERANDS}
```

IBM Software Hub user interface does not load after restoring

Fixed in: 5.2.1

Applies to: 5.2.0

Applies to: All backup and restore methods

Diagnosing the problem

The IBM Software Hub user interface doesn't start after a restore. When you try to log in, you see a blank screen.

Cause of the problem

The following pods fail and must be restarted to load the IBM Software Hub user interface:

platform-auth-service
platform-identity-management
platform-identity-provider

Resolving the problem

Run the following command to restart the pods:

for po in $(oc get po -l icpdsupport/module=im -n ${PROJECT_CPD_INST_OPERANDS} --no-headers | awk '{print $1}' | grep -v oidc); do oc delete po -n ${PROJECT_CPD_INST_OPERANDS} ${po};done;

Restore to same cluster fails with `restore-cpd-volumes` error

Fixed in: 5.2.1

Applies to: 5.2.0

Applies to: All backup and restore methods

Diagnosing the problem

IBM Software Hub fails to restore, and you see an error similar to the following example:

error: DataProtectionPlan=cpd-offline-tenant/restore-service-orchestrated-parent-workflow, Action=restore-cpd-volumes (index=1)
error: expected restore phase to be Completed, received PartiallyFailed

Cause of the problem

The ibm-streamsets-static-content pod is mounted as read-only, which causes the restore to partially fail.

Resolving the problem

Before you back up, exclude the IBM StreamSets pod:

Exclude the ibm-streamsets-static-content pod:

oc label -l icpdsupport/addOnId=streamsets,app.kubernetes.io/name=ibm-streamsets-static-content velero.io/exclude-from-backup=true

Take another back up and follow the back up and restore procedure for your environment.

FoundationDB cluster is stuck in `stage_mirror: post-restore` state after restore from Online backup

Applies to: 5.2.2

Applies to: All backup and restore methods

Diagnosing the problem

After you complete a restore from an online backup, the FoundationDB cluster gets stuck in the post-restore stage with an InProgress status.

oc get fdbcluster wkc-foundationdb-cluster --namespace wkc  -o json | jq  .status
{
  "backup_status": {
    "backup_phase": "",
    "backup_stage": "post-restore",
    "cpdbr_backup_index": 0,
    "cpdbr_backup_name": "",
    "cpdbr_backup_start": 0,
    "cpdbr_init_time": 0,
    "fdb_last_up_time": 0,
    "fdb_up_start_time": 0,
    "last_completion_time": 0,
    "last_update_time": 0
  },
  "coordinators_status": {
    "last_query_time": 0,
    "last_update_time": 0
  },
  "fdbStatus": "InProgress",
  "restore_phase": "New Restore job created",
  "shutdownStatus": " ",
  "stage_mirror": "post-restore"
}

The restore operation fails, and you see the following error message in the restore logs:

Restoring backup to version: 125493581420
ERROR: Attempted to restore into a non-empty destination database
Fatal Error: Attempted to restore into a non-empty destination database

Cause of the problem

The issue is caused by a timing issue during the post-restore process. When the restore process completes, FoundationDB pods start up and lineage ingestion services (specifically wdp-kg-ingestion-service and wkc-data-lineage-service) also begin writing to the database before the FoundationDB restore job can lock it. This situation causes a race condition. It results in the database being non-empty when the FoundationDB restore job attempts to clear and restore data, which causes the restore operation to fail.

Resolving the problem

Use the following commands to put the wkc operator in maintenance mode.

cpd-cli manage update-cr --component=wkc --cpd_instance_ns=<OPERAND_NS> --patch='{"spec":{"ignoreForMaintenance":true}}'

Use the following commands to stop any lineage ingestion job:

oc scale deploy wdp-kg-ingestion-service --replicas=0 -n <namespace>
oc scale deploy wkc-data-lineage-service --replicas=0 -n <namespace>

Edit the FoundationDB custom resource (CR):
```
oc edit fdbcluster wkc-foundationdb-cluster -n <namespace>
```
1. Force a shut down of the FoundationDB cluster by setting spec.shutdown to "force".
2. Check that the post-restore tag is present in the CR.
Wait for all FoundationDB pods to stop.
Start the FoundationDB cluster again by editing the FoundationDB CR and changing spec.shutdown from "force" to "false".
Verify the restore job completes successfully by checking the logs. The job should not have any unclean database errors.

Use the following command to verify FoundationDB status reaches Completed status:

oc get fdbcluster wkc-foundationdb-cluster -n <namespace> -o json | jq .status.fdbStatus

Use the following commands to scale up the lineage ingestion services:

oc scale deploy wdp-kg-ingestion-service --replicas=1 -n <namespace>
oc scale deploy wkc-data-lineage-service --replicas=1 -n <namespace>

Verify that the wkc CR reaches Completed status.

Identity resources have file names that are too long

Applies to: 5.2.0 and later

Applies to: All backup and restore methods

Diagnosing the problem

During the backup validation phase, the backup fails, and you see an error that says the file name is too long, for example:

time=2025-10-07T14:53:35.524157Z level=debug msg=exit backupValidationService.validateBackup
Error: failed to extract downloaded backup file '/tmp/backup-resources-f762cb7f-fea7-4d3a-9e13-c793147f7012-20251007145334-data.tar.gz': 
open /tmp/backup-resources-f762cb7f-fea7-4d3a-9e13-c793147f7012-20251007145334-data-20251007145334/resources/identities.user.openshift.io/cluster/xxx:yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy.json: 
file name too long

Cause of the problem

The error occurs when extracting the backup archive. One or more OpenShift identity resources have very long names, for example identities.user.openshift.io in the error above. These names exceed the filesystem's 255 character filename limit. This typically occurs with LDAP-generated identities that include the full distinguished name (DN) in the resource name. When the backup validation process attempts to extract the backup archive and create files for these resources, the operation fails because the filename is too long.

Resolving the problem

Manually mark the problematic identity resources so that they are excluded from the backup:

Use the following command to exclude all identity resources before you take a backup:
```
oc get identities.user.openshift.io --no-headers | awk '{print $1}' | xargs -I {} sh -c "oc label identities.user.openshift.io {} velero.io/exclude-from-backup=true"
```
Note: You can use the same command even if you already tried to take a backup, and the backup failed.
Retry the backup.

IBM Match 360 backup fails with an error stating that there are no matches for elasticsearch.opencontent.ibm.com

Fixed in: 5.2.1

Applies to: 5.2.0

Applies to: All backup and restore methods

Diagnosing the problem

When you attempt to perform an offline backup of the IBM Match 360 service, the backup will fail with the following error message:

error performing op preBackupViaConfigHookRule for resource mdm (configmap=mdm-cpd-aux-br-cm): no matches for elasticsearch.opencontent.ibm.com/, Resource=elasticsearchclusters

Cause of the problem

The IBM Match 360 backup ConfigMap is configured incorrectly.

Resolving the problem

To resolve this problem, update the IBM Match 360 backup ConfigMap to correct the issue:

Get the IBM Match 360 operator version:

export mdm_operator_pod=`oc get pods -n ${PROJECT_CPD_INST_OPERATORS}|grep mdm-operator-controller-manager|awk '{print $1}'`
echo mdm_operator_version=`oc exec -it $mdm_operator_pod -n cpd-operator  -- ls /opt/ansible/roles  |awk '{print $1}'`

Patch the backup template by running the following command:

oc exec -it $mdm_operator_pod -n ${PROJECT_CPD_INST_OPERATORS} -- sed -i 's/elasticsearchclusters.elasticsearch.opencontent.ibm.com/clusters.opensearch.cloudpackopen.ibm.com/g' /opt/ansible/roles/<mdm_operator_version>/addon/templates/mdm-backup-restore-extension-configmap.yaml.j2

Complete a redundant update in the IBM Match 360 CR (mdm-cr) to trigger operator reconciliation.
```
oc patch mdm mdm-cr --type=merge -p '{"spec":{"onboard":{"timeout_seconds": "800" }}}' -n ${PROJECT_CPD_INST_OPERANDS}
```
Wait for mdm-cr to reconcile itself. Proceed to the next step when mdm-cr is in a Completed state.
Start the backup process.

Resource validation at the end of a backup fails with `OOMKilled` status

Applies to: 5.2.0 and later

Applies to: Backup and restore with IBM Fusion

Diagnosing the problem

When performing a backup with IBM Fusion, the backup fails during the resource validation phase at the end of the backup process. The IBM Fusion transaction manager logs indicate the following errors

Recipe failed
BMYBR0009
There was an error when processing the job in the Transaction Manager service. 
The underlying error was: 'Recipe executed. Fail Count=1, rollback=True, last failed command: "ExecHook/br-service-hooks/resource-validation"   Extracted error message: Success.Hook run in cpd-operators:cpdbr-tenant-service-5cb9-tsj ended with internal rc 1 indicating hook execution ended in failure. Extracted error message: Running command '/cpdbr-scripts/cpdbr-oadp backup validate --tenant-operator-namespace=cpd-operators --namespace=ibm-backup-restore --spec-version=2.0.0' failed with exception: (0)\nReason: Handshake status 500 Internal Server Error -+-+- {'content-length': '28', 'content-type': 'text/plain; charset=utf-8', 'date': 'Fri, 13 Feb 2026 09:20:36 GMT'} -+-+- b'container not found ("main")'\n.'.

When you investigate, you see that the resource validation was causing the cpdbr-tenant-service pod to restart with an OOMKilled (exit code 137) status.

Cause of the problem

The resource validation process requires more memory than the default memory limit that is allocated in the cpdbr-tenant-service deployment. When the memory limit is insufficient, it causes the pod to be killed by the Kubernetes OOM (Out of Memory) killer.

This issue occurs more frequently in environments with large numbers of resources being backed up, where the validation process needs to process and validate all backed-up resources.

Resolving the problem

Increase the memory limit for the cpdbr-tenant-service deployment before performing a backup with IBM Fusion.

Check the memory limit of the cpdbr-tenant-service deployment.

oc get deployment \
-n ${PROJECT_CPD_INST_OPERATORS} cpdbr-tenant-service \
-o jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}'

Run the following command to increase the memory limit for cpdbr-tenant-service from 1Gi to 4Gi:


oc patch deployment cpdbr-tenant-service \
-n ${PROJECT_CPD_INST_OPERATORS} \
--type='json' \
-p='[{ "op": "replace", "path": "/spec/template/spec/containers/0/resources/limits/memory", "value": "4Gi" }]'

The deployment will automatically restart the pod with the new memory limit.

Wait for the pod to be ready, and then try to take another backup.
You can check if the pod is ready by using the following command:
```
oc get pods -n cpd-operators | grep cpdbr-tenant-service
```

Db2 backup fails at the Hook: br-service hooks/pre-backup step

Applies to: 5.2.0 and later

Applies to: Backup and restore with IBM Fusion

Diagnosing the problem

In the cpdbr-oadp.log file, you see messages like in the following example:

time=<timestamp> level=info msg=podName: c-db2oltp-5179995-db2u-0, podIdx: 0, container: db2u, actionIdx: 0, commandString: ksh -lc 'manage_snapshots --action suspend --retry 3', command: [sh -c ksh -lc 'manage_snapshots --action suspend --retry 3'], onError: Fail, singlePodOnly: false, timeout: 20m0s func=cpdbr-oadp/pkg/quiesce.executeCommand file=/go/src/cpdbr-oadp/pkg/quiesce/ruleexecutor.go:767
time=<timestamp> level=info msg=cmd stdout:  func=cpdbr-oadp/pkg/quiesce.executeCommand file=/go/src/cpdbr-oadp/pkg/quiesce/ruleexecutor.go:823
time=<timestamp> level=info msg=cmd stderr: [<timestamp>] - INFO: Setting wolverine to disable
Traceback (most recent call last):
  File "/usr/local/bin/snapshots", line 33, in <module>
    sys.exit(load_entry_point('db2u-containers==1.0.0.dev1', 'console_scripts', 'snapshots')())
  File "/usr/local/lib/python3.9/site-packages/cli/snapshots.py", line 35, in main
    snap.suspend_writes(parsed_args.retry)
  File "/usr/local/lib/python3.9/site-packages/snapshots/snapshots.py", line 86, in suspend_writes
    self._wolverine.toggle_state(enable=False, message="Suspend writes")
  File "/usr/local/lib/python3.9/site-packages/utils/wolverine/wolverine.py", line 73, in toggle_state
    self._toggle_state(state, message)
  File "/usr/local/lib/python3.9/site-packages/utils/wolverine/wolverine.py", line 77, in _toggle_state
    self._cmdr.execute(f'wvcli system {state} -m "{message}"')
  File "/usr/local/lib/python3.9/site-packages/utils/command_runner/command.py", line 122, in execute
    raise CommandException(err)
utils.command_runner.command.CommandException: Command failed to run:ERROR:root:HTTPSConnectionPool(host='localhost', port=9443): Read timed out. (read timeout=15)

Cause of the problem

The Wolverine high availability monitoring process was in a RECOVERING state before the backup was taken.

Check the Wolverine status by running the following command:

wvcli system status

Example output:

ERROR:root:REST server timeout: https://localhost:9443/status
ERROR:root:Retrying Request: https://localhost:9443/status
ERROR:root:REST server timeout: https://localhost:9443/status
ERROR:root:Retrying Request: https://localhost:9443/status
ERROR:root:REST server timeout: https://localhost:9443/status
ERROR:root:Retrying Request: https://localhost:9443/status
ERROR:root:REST server timeout: https://localhost:9443/status
HA Management is RECOVERING at <timestamp>.

The Wolverine log file /mnt/blumeta0/wolverine/logs/ha.log shows errors like in the following example:

<timestamp> [ERROR] <MainProcess:11490> [wolverine.ha.loop(loop.py:414)] -  check_and_recover: unhealthy_dm_set = {('c-db2oltp-5179995-db2u-0', 'node')}
<timestamp> [ERROR] <MainProcess:11490> [wolverine.ha.loop(loop.py:416)] - (c-db2oltp-5179995-db2u-0, node) : not OK
<timestamp> [ERROR] <MainProcess:11490> [wolverine.ha.loop(loop.py:421)] -  check_and_recover: unhealthy_dm_names = {'node'}

Resolving the problem

Do the following steps:

Re-initialize Wolverine:
```
wvcli system init --force
```
Wait until the Wolverine status is RUNNING. Check the status by running the following command:
```
wvcli system status
```
Retry the backup.

`wd-discovery-opensearch-client` pod fails during backup

Fixed in: 5.2.1

Applies to: 5.2.0 and later

Applies to: Backup and restore with IBM Fusion

Diagnosing the problem

To diagnose the problem, complete the following steps:

Check the backup and restore log:

oc -n ${PROJECT_CPD_INST_OPERANDS} exec -it $(oc -n ${PROJECT_CPD_INST_OPERANDS} get pods -licpdsupport/app=br-service -oname) -- cat cpdbr-oadp.log

Look for an error similar to the following example:

[2025-05-23T22:43:31,094][WARN ][r.suppressed             ] [wd-discovery-opensearch-client-000] path: /_snapshot/cloudpak/cloudpak_snapshot_2025-05-23-22-43-31, params: {pretty=, repository=cloudpak, wait_for_completion=true, snapshot=cloudpak_snapshot_2025-05-23-22-43-31}
org.opensearch.transport.RemoteTransportException: [wd-discovery-opensearch-master-000][10.128.7.239:9300][cluster:admin/snapshot/create]
Caused by: org.opensearch.repositories.RepositoryException: [cloudpak] Could not read repository data because the contents of the repository do not match its expected state. This is likely the result of either concurrently modifying the contents of the repository by a process other 
than this cluster or an issue with the repository's underlying storage. The repository has been disabled to prevent corrupting its contents. To re-enable it and continue using it please remove the repository from the cluster and add it again to make the cluster recover the known state of the repository from its physical contents.

Cause of the problem

Snapshot data might be corrupted.

Resolving the problem

Complete the following steps:

Quiesce the OpenSearch cluster:

oc patch cluster wd-discovery-opensearch --type merge --patch '{"spec": {"quiesce": true}}'

Wait until OpenSearch pods are deleted. Run the following command to check the pods are deleted:

oc get pods -l app.kubernetes.io/managed-by=ibm-opensearch-operator,job-name!=wd-discovery-opensearch-snapshot-repo

You'll see output similar to the following example:

oc get pods -l app.kubernetes.io/managed-by=ibm-opensearch-operator,job-name!=wd-discovery-opensearch-snapshot-repo
wd-discovery-opensearch-client-000                             1/1     Terminating   0             18h
wd-discovery-opensearch-client-001                             1/1     Terminating   0             18h
wd-discovery-opensearch-data-000                               1/1     Terminating   0             18h
wd-discovery-opensearch-data-001                               1/1     Terminating   0             18h
wd-discovery-opensearch-master-000                             1/1     Terminating   0             18h
wd-discovery-opensearch-master-001                             1/1     Terminating   0             18h
wd-discovery-opensearch-master-002                             1/1     Terminating   0             18h

Unquiesce the OpenSearch cluster:

oc patch cluster -n zen wd-discovery-opensearch --type merge --patch '{"spec": {"quiesce": false}}'

Wait until OpenSearch pods are running:

oc get pods -l app.kubernetes.io/managed-by=ibm-opensearch-operator,job-name!=wd-discovery-opensearch-snapshot-repo

You'll see output similar to the following example:

oc get pods -l app.kubernetes.io/managed-by=ibm-opensearch-operator,job-name!=wd-discovery-opensearch-snapshot-repo
NAME                                 READY   STATUS    RESTARTS   AGE
wd-discovery-opensearch-client-000   1/1     Running   0          2m9s
wd-discovery-opensearch-client-001   1/1     Running   0          2m8s
wd-discovery-opensearch-data-000     1/1     Running   0          2m8s
wd-discovery-opensearch-data-001     1/1     Running   0          2m8s
wd-discovery-opensearch-master-000   1/1     Running   0          2m8s
wd-discovery-opensearch-master-001   1/1     Running   0          2m8s
wd-discovery-opensearch-master-002   1/1     Running   0          2m7s

Retry the backup.

IBM Match 360 backup fails if it takes longer than 20 minutes

Fixed in: 5.2.2

Applies to: 5.2.1

Applies to: Backup with IBM Fusion

Diagnosing the problem

When you attempt to perform a backup of the IBM Match 360 service, the backup fails for both offline and online strategies during the post-backup phase if it does not complete within the 20-minute timeout period.

Cause of the problem

The IBM Match 360 backup does not have a long enough post-backup hook timeout value configured by default.

Resolving the problem

Before starting the backup process, resolve this problem by updating the IBM Match 360 backup configuration. To configure a longer timeout value:

Patch the IBM Match 360 CR (mdm-cr) to update the offline backup timeout value:

oc patch mdm mdm-cr --type=merge -p '{"spec":{"backup_restore:":{"br_timeout":{"offline_backup":{"post_backup":{"br_status_update": "2400s" }}}}}}' -n ${PROJECT_CPD_INST_OPERANDS}

Patch the IBM Match 360 CR (mdm-cr) to update the online backup timeout value:

oc patch mdm mdm-cr --type=merge -p '{"spec":{"backup_restore:":{"br_timeout":{"online_backup":{"post_backup":{"br_status_update": "2400s" }}}}}}' -n ${PROJECT_CPD_INST_OPERANDS}

Wait for mdm-cr to reconcile itself. Proceed to the next step when mdm-cr is in a Completed state.
Start the backup process.

Backup service location is unavailable during backup

Applies to: 5.2.0 and later

Applies to: Backup with IBM Fusion

If your cluster is running Red Hat OpenShift Container Platform 4.19 with IBM Fusion 2.11 and OADP 1.5.1, then the backup service location might enter Unavailable state. To resolve this issue, upgrade OADP to version 1.5.2.

Backup validation fails with `tls: failed to verify certificate: x509` error

Fixed in : 5.2.2

Applies to: 5.2.0 and 5.2.1

Applies to: Backup with IBM Fusion

Diagnosing the problem

The backup fails during validation and you receive the following error message:

tls: failed to verify certificate: x509: certificate signed by unknown authority func=cpdbr-oadp/cmd.Execute

Cause of the problem

The backup storage location uses a self-signed certificate. IBM Fusion did no recognize the certificate authority, so the connection and the backup validation fail.

Resolving the problem

Run the following script to add the --insecure-skip-tls-verify flag to the backup validation command:

brServiceHooksIndex=$(oc get frcpe ibmcpd-tenant -n ${PROJECT_CPD_INST_OPERATORS} -o json | jq '.spec.hooks | map(.name == "br-service-hooks") | index(true)')
resourceValidationIndex=$(oc get frcpe ibmcpd-tenant -n ${PROJECT_CPD_INST_OPERATORS} -o json | jq '.spec.hooks[] | select(.name == "br-service-hooks") | .ops | map(.name == "resource-validation") | index(true)')
resourceValidationCmd=$(oc get frcpe ibmcpd-tenant -n ${PROJECT_CPD_INST_OPERATORS} -o jsonpath="{.spec.hooks[$brServiceHooksIndex].ops[$resourceValidationIndex].command}")
newResourceValidationCmd=$(echo $resourceValidationCmd " --insecure-skip-tls-verify")
echo "cmd was: $resourceValidationCmd"
echo "cmd now: $newResourceValidationCmd"
echo "press any key to continue to patch fusion recipe..."
read

oc patch frcpe ibmcpd-tenant -n ${PROJECT_CPD_INST_OPERATORS} --type json -p "[{\"path\": \"/spec/hooks/$brServiceHooksIndex/ops/$resourceValidationIndex/command\", \"value\": \"$newResourceValidationCmd\", \"op\": \"replace\"}]"

currentCmdFromRecipe=$(oc get frcpe ibmcpd-tenant -n ${PROJECT_CPD_INST_OPERATORS} -o jsonpath="{.spec.hooks[$brServiceHooksIndex].ops[$resourceValidationIndex].command}")
echo "current command now from parent-recipe = '$currentCmdFromRecipe'"
echo "note: please refrain to run this script more than once"
echo "to reset the fusion software hub recipe (if you need to after this modification), please use 'cpd-cli oadp generate plan fusion parent-recipe --tenant-operator-namespace=\${PROJECT_CPD_INST_OPERATORS} --verbose --log-level=debug' command as stated in the documentation https://www.ibm.com/docs/en/software-hub/5.2.x?topic=software-installing-configuring-backup-restore-fusion"

Watson Discovery fails after restore with `opensearch` and `post_restore` unverified components

Applies to: 5.2.0 and later

Applies to: Backup and restore with IBM Fusion

Diagnosing the problem

After you restore, Watson Discovery becomes stuck with the following components listed as unverifiedComponents:

unverifiedComponents:
- opensearch
- post_restore

Additionally, the OpenSearch client pod might show an unknown container status, similar to the following example:

NAME                                 READY   STATUS                    RESTARTS   AGE
wd-discovery-opensearch-client-000   0/1     ContainerStatusUnknown    0          11h

Cause of the problem

The post_restore component depends on the opensearch component being verified. However, the OpenSearch client pod is not running, which prevents verification and causes the restore process to stall.

Resolving the problem

Manually delete the OpenSearch client pod to allow it to restart:

$ oc delete -n ${PROJECT_CPD_INST_OPERANDS} pod wd-discovery-opensearch-client-000

After the pod is restarted and verified, the post_restore component should complete the verification process.

Watson Discovery fails after restore with `mcg` and `post_restore` unverified components

Fixed in: 5.2.1

Applies to: 5.2.0

Applies to: Backup and restore with IBM Fusion

Diagnosing the problem

After you restore, Watson Discovery becomes stuck with the following components listed as unverifiedComponents:

unverifiedComponents:
- ingestion
- mcg
- post_restore

Additionally, the OpenSearch client pod might show a failed container status, similar to the following example:

NAME                                 READY   STATUS                    RESTARTS   AGE
wd-discovery-s3-bucket-job   Failed   0/1   9d   9d

Cause of the problem

The post_restore component depends on the mcg component being verified. However, during the post restore process, the retry feature for the mcg verification job is disabled. If the job fails on its first attempt, it will not retry automatically, causing the verification process to fail.

Resolving the problem

Manually delete the wd-discovery-s3-bucket-job job to allow it to restart:

$ oc delete -n ${PROJECT_CPD_INST_OPERANDS} job wd-discovery-s3-bucket-job

After the job is restarted and verified, the post_restore component should complete the verification process.

Restore process stuck at `db2u-aux-ckpt-cm-child.db2ucluster-ready/readyState`

Applies to: 5.2.0 and later

Applies to: Restore with IBM Fusion

Diagnosing the problem

When you use IBM Fusion to restore to the same cluster, the restore process fails when it times out while trying to verify that the db2ucluster or db2uinstance resources are in Ready status. You might receive something similar to the following error message:

There was an error when processing the job in the Transaction Manager service. 
The underlying error was: 'Execution of workflow restore of recipe ibmcpd-tenant completed. 
Number of failed commands: 1, last failed command: 
"CheckHook/db2u-aux-ckpt-cm-child.db2ucluster-ready/readyState"'.

Cause of the problem

IBM Fusion fails to restore because of a combination of factors:

The db2 velero plugin is not present in IBM Fusion DPA.
The IBM Fusion check hook requires both db2uinstance and db2ucluster resources to be present, and it cannot skip checks when resources don't exist in the cluster

Workaround

You have two options to resolve the issue:

Update IBM Fusion from version 2.10 to either version 2.10 with hotfixes or 2.10.1 version, and then restart the backup and restore
Reconfigure DataProtectionApplication (DPA) and start a fresh backup and restore.
1. Configure DPA from the source cluster by following the process for creating a DPA custom resource in Installing and configuring software on the source cluster for backup and restore with IBM Fusion.
2. Use IBM Fusion to create another backup.
3. Start another restore from the new backup.

Restore process stuck at `zenextensions-patch-ckpt-cm` step

Applies to: 5.2.0 and later

Applies to: Backup and restore with IBM Fusion 2.10 (without hotfixes)

Diagnosing the problem

IBM Fusion fails to restore because the process gets stuck at the zenextensions-patch-ckpt-cm step. The restore fails at the following stage:

ExecHook/zenextensions-patch-ckpt-cm-zen-1-child.zenextensions-hooks/force-reconcile-zenextensions

Cause of the problem

The issue is when the IBM Fusion exec hook cannot handle trailing new spaces, and the restore process gets stuck waiting for zenextensions hooks to complete.

Workarounds

You have two options to resolve the issue:

Update IBM Fusion from version 2.10 to version either 2.10 with hotfixes or 2.10.1 version, and then restart the backup and restore
Use the fusion-resume-restore script.

You can use the fusion-resume-restore.sh script to continue the restore process from the point where it failed.

Check that you have the prerequisites to run the script:
- yq 4.42.1 or later. You can download and install it from: https://github.com/mikefarah/yq.
- jq-1.7 or later
Download the fusion-resume-restore script from https://github.com/IBM/cpd-cli/blob/master/cpdops/files/fusion-resume-restore.sh.
Ensure you have write access to /tmp/fusion-resume-restore.
Run the script from hub:
```
./fusion-resume-restore.sh <fusion-restore-name> ${PROJECT_CPD_INST_OPERATORS}
```
The script displays the steps required to restore and their sequence numbers.
When the script asks you to Enter the key of the workflow hook/group to resume from, select the index number that excludes the zenextensions-patch-ckpt-cm step.
For example:
```
Enter the key of the workflow hook/group to resume from (0-101): 97
Selected workflow (index=97): "hook: zenextensions-patch-ckpt-cm-zen-1-child.zenextensions-hooks/force-reconcile-zenextensions"
```
When the script asks Resume the Fusion restore now?, choose a response based on whether you are restoring to the same or a different cluster.
- If restoring to the same cluster, reply y.
- If restoring to a different cluster, reply n.
  The script provides instructions to manually apply the CRs to resume the restore.

IBM Software Hub resources are not migrated

Applies to: 5.2.0 and later

Applies to: Portworx asynchronous disaster recovery

Diagnosing the problem

When you use Portworx asynchronous disaster recovery, the migration finishes almost immediately, and no volumes or the expected number of resources are migrated. Run the following command:

storkctl get migrations -n ${PX_ADMIN_NS}

Tip: ${PX_ADMIN_NS} is usually kube-system.

Example output:

NAME                                                CLUSTERPAIR       STAGE   STATUS       VOLUMES   RESOURCES   CREATED               ELAPSED                       TOTAL BYTES TRANSFERRED
cpd-tenant-migrationschedule-interval-<timestamp>   mig-clusterpair   Final   Successful   0/0       0/0         <timestamp>   Volumes (0s) Resources (3s)   0

Cause of the problem

This problem occurs starting with stork 23.11.0. Backup exec rules are not run, and expected IBM Software Hub resources are not migrated.

Resolving the problem

To resolve the problem, downgrade stork to a version prior to 23.11.0. For more information about stork releases, see the stork Releases page.

Scale down the Portworx operator so that it doesn't reset manual changes to the stork deployment:
```
oc scale -n ${PX_ADMIN_NS} deploy portworx-operator --replicas=0
```
Edit the stork deployment image version to a version prior to 23.11.0:
```
oc edit deploy -n ${PX_ADMIN_NS} stork
```
If you need to scale up the Portworx operator, run the following command.
Note: The Portworx operator will undo changes to the stork deployment and return to the original stork version.
```
oc scale -n ${PX_ADMIN_NS} deploy portworx-operator --replicas=1
```

Prompt tuning fails after restoring watsonx.ai

Applies to: 5.2.2 and later

Applies to: Backup and restore with the OADP utility

Diagnosing the problem

When you try to create a prompt tuning experiment, you see the following error message:

An error occurred while processing prompt tune training.

Resolving the problem

Do the following steps:

Restart the caikit operator:
```
oc rollout restart deployment caikit-runtime-stack-operator -n ${PROJECT_CPD_INST_OPERATORS}
```
Wait at least 2 minutes for the cais fmaas custom resource to become healthy.
Check the status of the cais fmaas custom resource by running the following command:
```
oc get cais fmaas -n ${PROJECT_CPD_INST_OPERANDS}
```
Retry the prompt tuning experiment.

Custom foundation models missing after backup and restore

Applies to: 5.2.0 and later

Fixed in: 5.2.2 (restart still required)

Applies to: Offline backup with the OADP utility

Diagnosing the problem

When performing offline backup and restore, BYOM (Bring Your Own Model) custom foundation model deployments are not included in the backup, and they do not exist after the restore. Additionally, inference on BYOM deployments fails after restore even when the deployment metadata is restored.

Cause of the problem

The issue occurs because the watsonx-ai-ifm operator's offline backup hook deletes all inferenceservices without excluding BYOM custom foundation models. The models are deleted during backup preparation, and they are not captured in the backup.

Resolving the problem

To resolve the issue, you must modify the deletion command before taking a backup. And you must restart the caikit-runtime-stack-operator pod after you complete the backup and restore process.

Note: You must restart the caikit-runtime-stack-operator pod after restoring for all 5.2.x releases, including 5.2.2.

Before taking a backup

Before taking an offline backup, you must do the following steps:

Note: If the operator pod is deleted or restarted, all these changes are lost. These steps must be repeated every time a new operator pod is created.

Put the watsonxaiifm operator and custom resource (CR) into maintenance mode:

oc patch watsonxaiifm watsonxaiifm-cr --namespace $PROJECT_INSTANCE_NAMESPACE --type=merge --patch '{"spec": {"forceReconcile":"true"}}'

Switch to the operator namespace:
```
oc project $PROJECT_OPERATOR_NAMESPACE
```

Access the watsonx-ai-ifm operator pod:

oc rsh ibm-cpd-watsonx-ai-ifm-operator-xxxxxxx

Navigate to the templates directory:

cd /opt/ansible/11.1.0/roles/watsonxaiifm-post-install/templates

Backup the original file:

cp post-cpbr-watsonxaiifm-inference-services-br-config.yaml.j2 \
   post-cpbr-watsonxaiifm-inference-services-br-config.yaml.j2.bak

Modify the deletion command to exclude custom foundation models:

sed -i 's|command:  \["/bin/bash", "-c", "kubectl delete inferenceservices -l '\''icpdsupport/addOnId=watsonx_ai_ifm'\'' ANDAND kubectl delete inferenceservices -l '\''syom_model'\''"\]|command: ["/bin/bash", "-c", "kubectl delete inferenceservices -l '\''icpdsupport/addOnId=watsonx_ai_ifm,model_type!=custom_foundation_model,type!=cfm'\'' -n $NAMESPACE ANDAND kubectl delete inferenceservices -l '\''syom_model,model_type!=custom_foundation_model,type!=cfm'\'' -n $NAMESPACE"]|' \
post-cpbr-watsonxaiifm-inference-services-br-config.yaml.j2

Verify the change:

grep -n "kubectl delete inferenceservices" \
post-cpbr-watsonxaiifm-inference-services-br-config.yaml.j2

Exit the container: Ctrl + D

Remove the operator and CR from maintenance mode:

oc patch watsonxaiifm watsonxaiifm-cr --namespace $PROJECT_INSTANCE_NAMESPACE --type=merge --patch '{"spec": {"forceReconcile":"false"}}'

After completing a restore

After you complete the offline restore, restart the caikit-runtime-stack-operator pod.

Switch to operator namespace:
```
oc project $PROJECT_OPERATOR_NAMESPACE
```
List and identify the caikit operator pod:
```
oc get pod|grep caikit
```
Delete the caikit-runtime-stack-operator pod to trigger restart:
```
oc delete pod caikit-runtime-stack-operator-9dcc859d8-bmz8q
```
Verify fmaas pods are running in the instance namespace:
```
oc project cpd-instance
oc get pod|grep fmaas
```
Confirm the following pods are running:
- fmaas-caikit-inf-prompt-tunes
- fmaas-caikit-trainer
- fmaas-mt
- fmaas-router

Restic backup that contains dynamically provisioned volumes in Amazon Elastic File System fails during restore

Applies to: 5.2.0 and later

Applies to: Backup and restore with the OADP utility

Diagnosing the problem

When trying to restore from an offline backup in Amazon Elastic File System (EFS), the restore process fails in the volume restore phase, and you might see something similar to the following error:

Status: Failed
Errors: 8
Warnings: 640

Action Errors:
DPP_NAME: cpd-offline-tenant/restore-service-orchestrated-parent-workflow
INDEX: 6
ACTION: restore-cpd-volumes
ERROR: error: expected restore phase to be Completed, received PartiallyFailed

In the Velero logs, you might see something similar to the following error:

time="2025-12-04T18:14:44Z" level=error msg="Restic command fail with ExitCode: 1. 
Process ID is 2077, Exit error is: exit status 1" 
PodVolumeRestore=openshift-adp/cpd-tenant-vol-r-00xda-x44x-4xx0-a9b1 
controller=PodVolumeRestore
stderr=ignoring error for /index-16: lchown /host_pods/.../mount/index-16: 
operation not permitted

Cause of the problem

Restic backups with EFS are not supported by Velero. Restic cannot properly restore file ownership and permissions on EFS volumes because it uses a different protocol for ownership operations.

Resolving the issue

If you want to use EFS for dynamically provisioned volumes, you must use Kopia backups instead of Restic. The OADP DataProtectionApplication (DPA) must use uploaderType: kopia.

Restoring Data Virtualization fails with metastore not running or failed to connect to database error

Applies to: 5.2.2 and later

Applies to: Online backup and restore with the OADP utility

Diagnosing the problem

View the status of the restore by running the following command:

cpd-cli oadp tenant-restore status ${TENANT_BACKUP_NAME}-restore --details

The output shows errors like in the following examples:

time=<timestamp>  level=INFO msg=Verifying if Metastore is listening
SERVICE              HOSTNAME                               NODE      PID  STATUS                                                                                                                                        	
Standalone Metastore c-db2u-dv-hurricane-dv                   -        -   Not running

time=<timestamp>  level=ERROR msg=Failed to connect to BigSQL database                                                                                                                                                                                                                        	     	                      	                                                                                                                                                                                                                                                                                                                                                         	                     	                                                                                                                                                                                                                                                                                                                                                         	
* error performing op postRestoreViaConfigHookRule for resource dv (configmap=cpd-dv-aux-ckpt-cm): 1 error occurred:                                                                                                                                                                                                                                     	
* error executing command su - db2inst1 -c '/db2u/scripts/bigsql-exec.sh /usr/ibmpacks/current/bigsql/bigsql/bigsql-cli/BIGSQL/package/scripts/bigsql-db2ubar-hook.sh -H POST -M RESTORE -L' (container=db2u podIdx=0 podName=c-db2u-dv-db2u-0 namespace=<namespace-name> auxMetaName=dv-aux component=dv actionIdx=0): command terminated with exit code 1

Cause of the problem

A timing issue causes restore posthooks to fail at the step where the posthooks check for the results of the db2 connect to bigsql command. The

db2
              connect to bigsql

command has failed because bigsql is restarting at around the same time.

Resolving the problem

Run the following command:

export CPDBR_ENABLE_FEATURES=experimental
cpd-cli oadp tenant-restore create ${TENANT_RESTORE_NAME}-cont \
--from-tenant-backup ${TENANT_BACKUP_NAME} \
--verbose \
--log-level debug \
--start-from cpd-post-restore-hooks

Backup fails on upgraded cluster due to EDB Postgres Enterprise ConfigMap timeout error

Applies to: 5.2.0 and later

Applies to: Offline backup and restore methods

Diagnosing the problem

After upgrading watsonx.data from version 5.1.x to 5.2.x, you performed an offline backup. However, the backup fails, and the backup pre-check hooks have timeout errors for EDB Postgres Enterprise ConfigMaps:

pre-backup (2):
  COMPONENT            CONFIGMAP            METHOD  STATUS  DURATION
  cpdbr-edb-patch-br   edb-patch-aux-br-cm  rule    error   6m0.025738359s
  lh-edb-br-component  lh-edb-aux-br-cm     rule    error   1m20.851377472s

Additionally, one of the EDB Postgres Enterprise pods (edb-2) might also get stuck with the following message:

Instance could be down, will not proceed with the reconciliation loop

Cause of the problem

During the upgrade, the specification for the postgresql PVC storage size changes from 10G to 10Gi, and the resizeInUseVolume flag gets set to true. However, the underlying storage class has AllowVolumeExpansion set to false, so the PostgreSQL operator enters a continuous reconciliation loop, where the operator repeatedly tries and fails to resize the PVC. You might see the following error:

persistentvolumeclaims "ibm-lh-postgres-edb-1" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

Although the PostgreSQL CR shows a Completed status, the operator is actually stuck in the resize loop and does not properly update the CR status. This prevents the backup pre-check hooks from completing, as they cannot refresh the backup instance annotation while the operator is in this unhealthy state.

Resolving the problem

The workaround to use depends on whether the storage class supports volume expansion.

Can enable volume expansion for storage class

Enable volume expansion by setting AllowVolumeExpansion to true in the storage class.
Allow the PostgreSQL operator to complete the PVC resize operation.
Verify the PostgreSQL cluster reaches Completed status.
Retry the backup operation.

Cannot enable volume expansion for storage class

You must manually recreate the PostgreSQL PVCs one at a time:

Delete one PostgreSQL pod and its associated PVC.
1. Wait for the new pod and PVC to be created.
2. Wait for PostgreSQL to sync up and stabilize.
Repeat the previous step for each of the remaining PostgreSQL pods. Delete only one pod at a time.
Verify all PVCs have been refreshed with the correct size.
Verify the PostgreSQL cluster is healthy.
Retry the backup operation.

Offline backup fails with `PartiallyFailed` error

Applies to: 5.2.2 and later

Applies to: Offline backup and restore with the OADP utility

Diagnosing the problem

In the Velero logs, you see errors like in the following example:

time="<timestamp>" level=error msg="Current BackupStorageLocations available/unavailable/unknown: 0/0/1)" controller=backup-storage-location logSource="/remote-source/velero/app/pkg/controller/backup_storage_location_controller.go:180"
time="<timestamp>" level=error msg="error encountered while scanning stdout" backupLocation=oadp-operator/dpa-sample-1 cmd=/plugins/velero-plugin-for-aws controller=backup-sync error="read |0: file already closed" logSource="/remote-source
/velero/app/pkg/plugin/clientmgmt/process/logrus_adapter.go:90"
time="<timestamp>" level=error msg="Restic command fail with ExitCode: 1. Process ID is 906, Exit error is: exit status 1" logSource="/remote-source/velero/app/pkg/util/exec/exec.go:66"
time="<timestamp>" level=error msg="pod volume backup failed: data path backup failed: error running restic backup command restic backup --repo=s3:http://minio-velero.apps.jctesti23.examplehost.example.com/velero/cpdbackup/restic/cpd-instance --pa
ssword-file=/tmp/credentials/oadp-operator/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=pod-uid=1ed9d52f-2f6d-4978-930a-4d8e30acced1 --tag=pvc-uid=76b76bc4-27d4-4369-886c-1272dfdf9ce9 --tag=volume=cc-home-p
vc-vol --tag=backup=cpd-tenant-vol-485eef74-efbe-11ef-b2bd-00000a0b44c3 --tag=backup-uid=b55d6323-9875-4afe-b605-646250cbd55c --tag=ns=cpd-instance --tag=pod=cpdbr-vol-mnt --host=velero --json with error: exit status 3 stderr: {\"message_type\":\"e
rror\",\"error\":{},\"during\":\"scan\",\"item\":\"/host_pods/1ed9d52f-2f6d-4978-930a-4d8e30acced1/volumes/kubernetes.io~nfs/pvc-76b76bc4-27d4-4369-886c-1272dfdf9ce9/.scripts/system\"}\n{\"message_type\":\"error\",\"error\":{},\"during\":\"archival
\",\"item\":\".scripts/system\"}\n{\"message_type\":\"error\",\"error\":{\"Op\":\"open\",\"Path\":\"_global_/security/artifacts/metakey\",\"Err\":13},\"during\":\"archival\",\"item\":\"/host_pods/1ed9d52f-2f6d-4978-930a-4d8e30acced1/volumes/kuberne
tes.io~nfs/pvc-76b76bc4-27d4-4369-886c-1272dfdf9ce9/_global_/security/artifacts/metakey\"}\nWarning: at least one source file could not be read\n" backup=oadp-operator/cpd-tenant-vol-485eef74-efbe-11ef-b2bd-00000a0b44c3 logSource="/remote-source/ve
lero/app/pkg/podvolume/backupper.go:328"
time="<timestamp>" level=error msg="pod volume backup failed: data path backup failed: error running restic backup command restic backup --repo=s3:http://minio-velero.apps.jctesti23.cp.fyre.ibm.com/velero/cpdbackup/restic/cpd-instance --pa
ssword-file=/tmp/credentials/oadp-operator/velero-repo-credentials-repository-password --cache-dir=/scratch/.cache/restic . --tag=pod=cpdbr-vol-mnt --tag=pod-uid=1ed9d52f-2f6d-4978-930a-4d8e30acced1 --tag=pvc-uid=93e9e23c-d80a-49cc-80bb-31a36524e0d
c --tag=volume=data-rabbitmq-ha-0-vol --tag=backup=cpd-tenant-vol-485eef74-efbe-11ef-b2bd-00000a0b44c3 --tag=backup-uid=b55d6323-9875-4afe-b605-646250cbd55c --tag=ns=cpd-instance --host=velero --json with error: exit status 3 stderr: {\"message_typ
e\":\"error\",\"error\":{\"Op\":\"open\",\"Path\":\".erlang.cookie\",\"Err\":13},\"during\":\"archival\",\"item\":\"/host_pods/1ed9d52f-2f6d-4978-930a-4d8e30acced1/volumes/kubernetes.io~nfs/pvc-93e9e23c-d80a-49cc-80bb-31a36524e0dc/.erlang.cookie\"}
\nWarning: at least one source file could not be read\n" backup=oadp-operator/cpd-tenant-vol-485eef74-efbe-11ef-b2bd-00000a0b44c3 logSource="/remote-source/velero/app/pkg/podvolume/backupper.go:328"

Cause of the problem

The restic folder was deleted after backups were cleaned up (deleted). This problem is a Velero known issue. For more information, see velero does not recreate restic|kopia repository from manifest if its directories are deleted on s3.

Resolving the problem

Do the following steps:

Get the list of backup repositories:

oc get backuprepositories -n ${OADP-OPERATOR-NAMESPACE} -o yaml

Check for old or invalid object storage URLs.
Check that the object storage path is in the backuprepositories custom resource.
Check that the <objstorage>/<bucket>/<prefix>/restic/<namespace>/config file exists.
If the file does not exist, make sure that you do not share the same <objstorage>/<bucket>/<prefix> with another cluster, and specify a different <prefix>.
Delete backup repositories that are invalid for the following reasons:
- The path does not exist anymore in the object storage.
- The restic/<namespace>/config file does not exist.
```
oc delete backuprepositories -n ${OADP_OPERATOR_NAMESPACE} <backup-repository-name>
```

Backup fails after a service is upgraded and then uninstalled

Applies to: 5.2.2 and later

Applies to: Backup and restore with the OADP utility

Diagnosing the problem

The problem occurs after a service was upgraded from IBM Cloud Pak® for Data 4.8.x to IBM Software Hub 5.2.1 and later uninstalled. When you try to take a backup by running the cpd-cli oadp tenant-backup command, the backup fails. In the CPD-CLI*.log file, you see an error message like in the following example:

Error: global registry check failed: 1 error occurred:
        * error from addOnId=watsonx_ai: 2 errors occurred:
        * failed to find aux configmap 'cpd-watsonxai-maint-aux-ckpt-cm' in tenant service namespace='<namespace_name>': : configmaps "cpd-watsonxai-maint-aux-ckpt-cm" not found
        * failed to find aux configmap 'cpd-watsonxai-maint-aux-br-cm' in tenant service namespace='<namespace_name>': : configmaps "cpd-watsonxai-maint-aux-br-cm" not found




[ERROR] <timestamp> RunPluginCommand:Execution error:  exit status 1

Resolving the problem

Re-run the backup command with the --skip-registry-check option. For example:

cpd-cli oadp tenant-backup create ${TENANT_OFFLINE_BACKUP_NAME} \
--namespace ${OADP_PROJECT} \
--vol-mnt-pod-mem-request=1Gi \
--vol-mnt-pod-mem-limit=4Gi \
--tenant-operator-namespace ${PROJECT_CPD_INST_OPERATORS} \
--mode offline \
--skip-registry-check \
--image-prefix=registry.redhat.io/ubi9 \
--log-level=debug \
--verbose &> ${TENANT_OFFLINE_BACKUP_NAME}.log&

Offline backup fails after watsonx.ai is uninstalled

Applies to: 5.2.0 and later

Applies to: Offline backup and restore with the OADP utility

Diagnosing the problem

The problem occurs when you try to take an offline backup after watsonx.ai was uninstalled. The backup process fails when post-backup hooks are run. In the CPD-CLI*.log file, you see error messages like in the following example:

time=<timestamp> level=info msg=cmd stderr: <timestamp> [emerg] 233346#233346: host not found in upstream "wx-inference-proxyservice:18888" in /nginx_data/extensions/upstreams/latest-510_watsonxaiifm-routes_ie_226.conf:10
nginx: [emerg] host not found in upstream "wx-inference-proxyservice:18888" in /nginx_data/extensions/upstreams/latest-510_watsonxaiifm-routes_ie_226.conf:10
nginx: configuration file /usr/local/openresty/nginx/conf/nginx.conf test failed
 func=cpdbr-oadp/pkg/quiesce.executeCommand file=/a/workspace/oadp-upload/pkg/quiesce/ruleexecutor.go:824
time=<timestamp> level=warning msg=failed to get exec hook JSON result for container=ibm-nginx-container podIdx=0 podName=ibm-nginx-fd79d5686-cdpnj namespace=latest-510 auxMetaName=lite-maint-aux component=lite-maint actionIdx=0 err=could not find JSON exec hook result func=cpdbr-oadp/pkg/quiesce.executeCommand file=/a/workspace/oadp-upload/pkg/quiesce/ruleexecutor.go:852
time=<timestamp> level=warning msg=no exec hook JSON result found for container=ibm-nginx-container podIdx=0 podName=ibm-nginx-fd79d5686-cdpnj namespace=latest-510 auxMetaName=lite-maint-aux component=lite-maint actionIdx=0 func=cpdbr-oadp/pkg/quiesce.executeCommand file=/a/workspace/oadp-upload/pkg/quiesce/ruleexecutor.go:855
time=<timestamp> level=info msg=exit executeCommand func=cpdbr-oadp/pkg/quiesce.executeCommand file=/a/workspace/oadp-upload/pkg/quiesce/ruleexecutor.go:860
time=<timestamp> level=error msg=command terminated with exit code 1

Cause of the problem

After watsonx.ai is uninstalled, the nginx configuration in the ibm-nginx pod is not cleared up, and the pod fails.

Resolving the problem

Restart all ibm-nginx pods.

oc delete pod \
-n=${PROJECT_CPD_INST_OPERANDS} \
-l component=ibm-nginx

Db2 Big SQL backup pre-hook and post-hook fail during offline backup

Applies to: 5.2.0 and later

Applies to: Offline backup and restore with the OADP utility

Diagnosing the problem

In the db2diag logs of the Db2 Big SQL head pod, you see error messages such as in the following example when backup pre-hooks are running:

<timestamp>          LEVEL: Event
PID     : 3415135              TID : 22544119580160  PROC : db2star2
INSTANCE: db2inst1             NODE : 000
HOSTNAME: c-bigsql-<xxxxxxxxxxxxxxx>-db2u-0
FUNCTION: DB2 UDB, base sys utilities, DB2StartMain, probe:5692
MESSAGE : ZRC=0xFFFFFBD0=-1072
          SQL1072C  The request failed because the database manager resources
          are in an inconsistent state. The database manager might have been
          incorrectly terminated, or another application might be using system
          resources in a way that conflicts with the use of system resources by
          the database manager.

Cause of the problem

The Db2 database was unable to start because of the error code SQL1072C. As a result, the bigsql start command that runs as part of the post-backup hook hangs, which produces the timeout of the post-hook. The post-hook cannot succeed unless Db2 is brought back to a stable state and the bigsql start command runs successfully. The Db2 Big SQL instance is left in an unstable state.

Resolving the problem

Do one or both of the following troubleshooting and cleanup procedures.

Tip: For more information about the SQL1072C error code and how to resolve it, see SQL1000-1999 in the Db2 documentation.

Remove all the database manager processes running under the Db2 instance ID

Do the following steps:

oc -n ${PROJECT_CPD_INST_OPERANDS} rsh $(oc -n ${PROJECT_CPD_INST_OPERANDS} | grep -i c-bigsql | grep -i db2u-0 | cut -d' ' -f 1) bash

Switch to the db2inst1 user:
```
su - db2inst1
```
List all the database manager processes that are running under db2inst1:
```
db2_ps
```
Remove these processes:
```
kill -9 <process-ID>
```

Ensure that no other application is running under the Db2 instance ID, and then remove all resources owned by the Db2 instance ID

Do the following steps:

oc -n ${PROJECT_CPD_INST_OPERANDS} rsh $(oc -n ${PROJECT_CPD_INST_OPERANDS} | grep -i c-bigsql | grep -i db2u-0 | cut -d' ' -f 1) bash

Switch to the db2inst1 user:
```
su - db2inst1
```
List all IPC resources owned by db2inst1:
```
ipcs | grep db2inst1
```
Remove these resources:
```
ipcrm -[q|m|s] db2inst1
```

OpenSearch operator fails during backup

Applies to: 5.2.0 and later

Applies to: Offline backup and restore with the OADP utility

Diagnosing the problem

After restoring, verify the list of restored indices in OpenSearch. If only indices that start with a dot, for example, .ltstore, are present, and expected non-dot-prefixed indices are missing, this indicates the issue.

Cause of the problem

During backup and restore, the OpenSearch operator restores indices that start with either "." or indices that do not start with ".", but not both. This behavior affects Watson Discovery deployments where both types of indices are expected to be restored.

Resolving the problem

Complete the following steps to resolve the issue:

Access the client pod:

oc rsh -n ${PROJECT_CPD_INST_OPERANDS} wd-discovery-opensearch-client-000

Set the credentials and repository:

user=$(find /workdir/internal_users/ -mindepth 1 -maxdepth 1 | head -n 1 | xargs basename)
password=$(cat "/workdir/internal_users/${user}")
repo="cloudpak"

Get the latest snapshot:

last_snapshot=$(curl --retry 5 --retry-delay 5 --retry-all-errors -k -X GET "https://${user}:${password}@${OS_HOST}/_cat/snapshots/${repo}?h=id&s=end_epoch" | tail -n1)

Check that the latest snapshot was saved:
```
echo $last_snapshot
```

Restore the snapshot:

curl -k -X POST "https://${user}:${password}@${OS_HOST}/_snapshot/${repo}/${last_snapshot}/_restore?wait_for_completion=true" \
  -d '{"indices": "-.*","include_global_state": false}' \
  -H 'Content-Type: application/json'

This command can take a while to run before output is shown. After it completes, you'll see output similar to the following example:

{
    "snapshot": {
        "snapshot": "cloudpak_snapshot_2025-06-10-13-30-45",
        "indices": [
            "966d1979-52e8-6558-0000-019759db7bdc_notice",
            "b49b2470-70c3-4ba1-9bd2-a16d72ffe49f_curations",
			...
        ],
        "shards": {
            "total": 142,
            "failed": 0,
            "successful": 142
        }
    }
}

This restores all indices, including both dot-prefixed and non-dot-prefixed ones.

Security issues

Security scans return an `Inadequate Account Lockout Mechanism` message

Applies to: 5.2.0 and later

Diagnosing the problem

If you run a security scan against IBM Software Hub, the scan returns the following message.

Inadequate Account Lockout Mechanism

Resolving the problem

This is by design. It is strongly recommended that you use an enterprise-grade password management solution, such as SAML SSO or an LDAP provider for password management, as described in the following resources.

The Kubernetes version information is disclosed

Applies to: 5.2.0 and later

Diagnosing the problem

If you run an Aqua Security scan against your cluster, the scan returns the following issue:

KHV002 - Kubernetes issue disclosure

Resolving the problem

This is expected based on the following solution document from Red Hat OpenShift Customer Portal: Hide kubernetes /version API endpoint in OpenShift Container Platform 4.

Known issues and limitations for IBM Software Hub

Customer-reported issues

General issues

After rebooting a cluster that uses OpenShift Data Foundation storage, some IBM Software Hub services aren't functional

Auxiliary modules are not found when running export-import commands on ppc64le hardware

Export jobs are not deleted if the export specification file includes incorrectly formatted JSON

The health service-functionality check fails at certain API endpoints for AI Factsheets, Data Product Hub, OpenPages, and watsonx.data

The health service-functionality check fails on the v2/ingestion_jobs API endpoint for watsonx.data

Cannot scale down IBM Knowledge Catalog

Installation and upgrade issues

The setup-instance command fails waiting for the ibm-namespace-scope-operator operator to be installed or upgraded

The setup-instance command fails if the ibm-common-service-operator-service service is not found

The Switch locations icon is not available if the apply-cr command times out

Upgrades fail if the Data Foundation Rook Ceph cluster is unstable

After you upgrade a Red Hat OpenShift Container Platform cluster, the FoundationDB resource can become unavailable

Persistent volume claims with the WaitForFirstConsumer volume binding mode are flagged by the installation health checks

Node pinning is not applied to postgresql pods

The ibm-nginx deployment does not scale fast enough when automatic scaling is configured

Uninstalling IBM watsonx services does not remove the IBM watsonx experience

Backup and restore issues

Issues that apply to several backup and restore methods

Backup and restore issues with the OADP utility

Backup and restore issues with IBM Fusion

Backup and restore issues with Portworx

ObjectBucketClaim is not supported by the OADP utility

After a restore, OperandRequest timeout error in the ZenService custom resource

After backing up or restoring IBM Match 360, Redis fails to return to a Completed state

Watson OpenScale fails after restore due to Db2 (db2oltp) or Db2 Warehouse (db2wh) configuration

Watson OpenScale fails after restore for some Watson Machine Learning deployments

IBM Software Hub user interface does not load after restoring

Restore to same cluster fails with restore-cpd-volumes error

FoundationDB cluster is stuck in stage_mirror: post-restore state after restore from Online backup

Identity resources have file names that are too long

IBM Match 360 backup fails with an error stating that there are no matches for elasticsearch.opencontent.ibm.com

Resource validation at the end of a backup fails with OOMKilled status

Db2 backup fails at the Hook: br-service hooks/pre-backup step

wd-discovery-opensearch-client pod fails during backup

IBM Match 360 backup fails if it takes longer than 20 minutes

Backup service location is unavailable during backup

Backup validation fails with tls: failed to verify certificate: x509 error

Watson Discovery fails after restore with opensearch and post_restore unverified components

Watson Discovery fails after restore with mcg and post_restore unverified components

Restore process stuck at db2u-aux-ckpt-cm-child.db2ucluster-ready/readyState

Restore process stuck at zenextensions-patch-ckpt-cm step

IBM Software Hub resources are not migrated

Prompt tuning fails after restoring watsonx.ai

Custom foundation models missing after backup and restore

Restic backup that contains dynamically provisioned volumes in Amazon Elastic File System fails during restore

Restoring Data Virtualization fails with metastore not running or failed to connect to database error

Backup fails on upgraded cluster due to EDB Postgres Enterprise ConfigMap timeout error

Offline backup fails with PartiallyFailed error

Backup fails after a service is upgraded and then uninstalled

Offline backup fails after watsonx.ai is uninstalled

Db2 Big SQL backup pre-hook and post-hook fail during offline backup

OpenSearch operator fails during backup

Security issues

Security scans return an Inadequate Account Lockout Mechanism message

The Kubernetes version information is disclosed

Auxiliary modules are not found when running `export-import` commands on ppc64le hardware

The health service-functionality check fails on the `v2/ingestion_jobs` API endpoint for watsonx.data

The `setup-instance` command fails waiting for the `ibm-namespace-scope-operator` operator to be installed or upgraded

The `setup-instance` command fails if the `ibm-common-service-operator-service` service is not found

The Switch locations icon is not available if the `apply-cr` command times out

Persistent volume claims with the `WaitForFirstConsumer` volume binding mode are flagged by the installation health checks

Node pinning is not applied to `postgresql` pods

The `ibm-nginx` deployment does not scale fast enough when automatic scaling is configured

After backing up or restoring IBM Match 360, Redis fails to return to a `Completed` state

Watson OpenScale fails after restore due to Db2 `(db2oltp)` or Db2 Warehouse `(db2wh)` configuration

Restore to same cluster fails with `restore-cpd-volumes` error

FoundationDB cluster is stuck in `stage_mirror: post-restore` state after restore from Online backup

Resource validation at the end of a backup fails with `OOMKilled` status

`wd-discovery-opensearch-client` pod fails during backup

Backup validation fails with `tls: failed to verify certificate: x509` error

Watson Discovery fails after restore with `opensearch` and `post_restore` unverified components

Watson Discovery fails after restore with `mcg` and `post_restore` unverified components

Restore process stuck at `db2u-aux-ckpt-cm-child.db2ucluster-ready/readyState`

Restore process stuck at `zenextensions-patch-ckpt-cm` step

Offline backup fails with `PartiallyFailed` error

Security scans return an `Inadequate Account Lockout Mechanism` message