StatefulSet error when restoring IBM Software Hub

Restoring IBM® Software Hub fails because the StatefulSet c-db2oltp-wkc-db2u is not ready.

Symptoms

In the log file, you see the following messages:

time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=error msg=failed to wait for statefulset c-db2oltp-wkc-db2u in namespace zen: timed out waiting for the condition func=cpdbr-oadp/pkg/kube.waitForStatefulSetPods file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:173
time=<timestamp> level=info msg=exit WaitForStatefulSetPodsViaLabels func=cpdbr-oadp/pkg/kube.KubeAPI.WaitForStatefulSetPodsViaLabels file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:155
time=<timestamp> level=info msg=exit waitForResourcePods func=cpdbr-oadp/pkg/planner.waitForResourcePods file=/go/src/cpdbr-oadp/pkg/planner/restore.go:571
time=<timestamp> level=info msg=exit RunRestoreHooks func=cpdbr-oadp/pkg/planner.RunRestoreHooks file=/go/src/cpdbr-oadp/pkg/planner/restore.go:522
time=<timestamp> level=error msg=restore post hooks error: timed out waiting for the condition func=cpdbr-oadp/cmd/cli/restore.(*createCommandContext).processCreate file=/go/src/cpdbr-oadp/cmd/cli/restore/create.go:557
restore post hooks error: timed out waiting for the condition
time=<timestamp> level=info msg=Successfully constructed k8s in cluster client func=cpdbr-oadp/pkg/kube.getKubernetesInClusterClient file=/go/src/cpdbr-oadp/pkg/kube/client.go:88
time=<timestamp> level=info msg=k8s server version: v1.25.8+37a9a08 func=cpdbr-oadp/pkg/kube.NewKubeAPI file=/go/src/cpdbr-oadp/pkg/kube/client.go:152
.
.
.
** PHASE [RESTORE CREATE/END] *************************************************
Error: restore failed with error: timed out waiting for the condition
time=<timestamp> level=error msg=restore failed with error: timed out waiting for the condition func=cpdbr-oadp/cmd.Execute file=/go/src/cpdbr-oadp/cmd/root.go:88
Time: <timestamp> level=error -  cpd-tenant-restore-<timestamp>-r4 failed

Causes

Outdated security context constraints (SCCs) exist in the cluster.

Environment

This problem occurs when the IBM Software Hub deployment includes services with a dependency on Db2U, such as:
  • Data Virtualization
  • Db2
  • Db2 Big SQL
  • Db2 Warehouse
  • OpenPages

Diagnosing the problem

Run the following command:
oc describe sts c-db2oltp-wkc-db2u
The command produces the following output:
...
Events:
  Type     Reason        Age                    From                    Message
  ----     ------        ----                   ----                    -------
  Warning  FailedCreate  4m32s (x186 over 17h)  statefulset-controller  create Pod c-db2oltp-wkc-db2u-0 in StatefulSet c-db2oltp-wkc-db2u failed error: pods "c-db2oltp-wkc-db2u-0" is forbidden: unable to validate against any security context constraint: ... spec.initContainers[0].securityContext.runAsUser: Invalid value: 500: must be in the ranges: ... Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Resolving the problem

Do the following steps.
  1. Delete the outdated SCCs:
    oc get scc | grep ${PROJECT_CPD_INST_OPERANDS} | awk '{print $1}' | xargs oc delete scc
  2. Delete IBM Software Hub projects.

    For more information, see IBM Software Hub online backup and restore to the same cluster with the OADP utility.

  3. Retry the restore.