StatefulSet error when restoring Cloud Pak for Data

Restoring Cloud Pak for Data fails because the StatefulSet c-db2oltp-wkc-db2u is not ready.

Symptoms

In the log file, you see the following messages:

time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=debug msg=waiting for replicas of c-db2oltp-wkc-db2u statefulset (0/1) in namespace zen func=cpdbr-oadp/pkg/kube.waitForStatefulSet.func1 file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:98
time=<timestamp> level=error msg=failed to wait for statefulset c-db2oltp-wkc-db2u in namespace zen: timed out waiting for the condition func=cpdbr-oadp/pkg/kube.waitForStatefulSetPods file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:173
time=<timestamp> level=info msg=exit WaitForStatefulSetPodsViaLabels func=cpdbr-oadp/pkg/kube.KubeAPI.WaitForStatefulSetPodsViaLabels file=/go/src/cpdbr-oadp/pkg/kube/statefulset.go:155
time=<timestamp> level=info msg=exit waitForResourcePods func=cpdbr-oadp/pkg/planner.waitForResourcePods file=/go/src/cpdbr-oadp/pkg/planner/restore.go:571
time=<timestamp> level=info msg=exit RunRestoreHooks func=cpdbr-oadp/pkg/planner.RunRestoreHooks file=/go/src/cpdbr-oadp/pkg/planner/restore.go:522
time=<timestamp> level=error msg=restore post hooks error: timed out waiting for the condition func=cpdbr-oadp/cmd/cli/restore.(*createCommandContext).processCreate file=/go/src/cpdbr-oadp/cmd/cli/restore/create.go:557
restore post hooks error: timed out waiting for the condition
time=<timestamp> level=info msg=Successfully constructed k8s in cluster client func=cpdbr-oadp/pkg/kube.getKubernetesInClusterClient file=/go/src/cpdbr-oadp/pkg/kube/client.go:88
time=<timestamp> level=info msg=k8s server version: v1.25.8+37a9a08 func=cpdbr-oadp/pkg/kube.NewKubeAPI file=/go/src/cpdbr-oadp/pkg/kube/client.go:152
.
.
.
** PHASE [RESTORE CREATE/END] *************************************************
Error: restore failed with error: timed out waiting for the condition
time=<timestamp> level=error msg=restore failed with error: timed out waiting for the condition func=cpdbr-oadp/cmd.Execute file=/go/src/cpdbr-oadp/cmd/root.go:88
Time: <timestamp> level=error -  cpd-tenant-restore-<timestamp>-r4 failed

Causes

Outdated security context constraints (SCCs) exist in the cluster.

Environment

This problem occurs when the Cloud Pak for Data deployment includes services with a dependency on Db2U, such as:
  • Db2®
  • Db2 Big SQL
  • Db2 Warehouse
  • Informix®
  • OpenPages®
  • Watson Query

Diagnosing the problem

Run the following command:
oc describe sts c-db2oltp-wkc-db2u
The command produces the following output:
...
Events:
  Type     Reason        Age                    From                    Message
  ----     ------        ----                   ----                    -------
  Warning  FailedCreate  4m32s (x186 over 17h)  statefulset-controller  create Pod c-db2oltp-wkc-db2u-0 in StatefulSet c-db2oltp-wkc-db2u failed error: pods "c-db2oltp-wkc-db2u-0" is forbidden: unable to validate against any security context constraint: ... spec.initContainers[0].securityContext.runAsUser: Invalid value: 500: must be in the ranges: ... Forbidden: not usable by user or serviceaccount, provider "privileged": Forbidden: not usable by user or serviceaccount]

Resolving the problem

Do the following steps.
  1. Delete the outdated SCCs:
    oc get scc | grep ${PROJECT_CPD_INST_OPERANDS} | awk '{print $1}' | xargs oc delete scc
  2. Delete Cloud Pak for Data projects.

    For more information, see Delete Cloud Pak for Data projects.

  3. Retry the restore.