Postgres cluster custom resource is not created during upgrade

When you upgrade to IBM Cloud Pak foundational services version 4.6 or later while using Postgres cluster as a database, the common-service-db Postgres cluster custom resource (CR) might not be successfully created. This issue might occur if you apply a patch for the license key of the embedded EDB PostgreSQL database on an old version of IBM Cloud Pak foundational services.

Important: This issue can occur when you upgrade to one of the following versions of IBM Cloud Pak foundational services: 4.6.2 to 4.6.7, 4.7.x, 4.8.x and 4.9.x. This issue is fixed in IBM Cloud Pak foundational services version 4.6.8, 4.10.0 and later. For more information, see Embedded EDB PostgreSQL database license key expires on October 1st, 2024 used for IBM Cloud Pak operators.

Symptom

• Upgrade does not complete and you can observe the following symptoms:

• After you upgrade IBM Cloud Pak foundational services, you cannot log in to cp-console or the IBM Cloud Pak console.

• There are no common-service-db Postgres cluster CRs created in the namespace where foundational services are installed. Run the following command to check for the common-service-db cluster CRs:

    oc get cluster.postgresql.k8s -n <namespace-where-foundational services-are-installed> | grep common-service-db
  

  You might see the empty output.

• The pods related to IM (platform-*) are stuck on Init status. Run the following command to check for all platform-* pods.

    oc get pods -n <namespace-where-foundational services-are-installed>  | grep "platform"
  

  In response you get a list of pods, all stuck in Init status.

    platform-auth-service-768bc7c449-6sjwc                            0/1     Init:0/1             25h
    platform-identity-management-7d6d97446f-x62qp                     0/1      Init:0/1 0             25h
    platform-identity-provider-9f5dfd8d4-lrh4h                        0/1      Init:0/1 0             25h
  

• You get the following error in the operand-deployment-lifecycle-manager pod logs. Run the following command to view the logs:

    oc logs -n <namespace-where-foundational services-operator-are-installed> -l name=operand-deployment-lifecycle-manager
  

  The following error is displayed in the logs:

    E0912 06:57:24.564354    1 manager.go:679] Failed to get value reference from Object cpd-instance1-operator/postgresql-operator-controller-manager-config with path .metadata.annotations.ibm-license-key-applied: failed to parse path .metadata.annotations.ibm-license-key-applied from Secret cpd-instance1-operator/postgresql-operator-controller-manager-config: ibm-license-key-applied is not found
    E0912 06:57:24.564437    1 manager.go:571] Failed to get value reference from Object for OperandConfig cpd-instance1/common-service on field description: failed to parse path .metadata.annotations.ibm-license-key-applied from Secret cpd-instance1-operator/postgresql-operator-controller-manager-config: ibm-license-key-applied is not found
    E0912 06:57:24.564450    1 reconcile_operand.go:529] Failed to parse value reference in resource cpd-instance1/common-service-db: failed to parse path .metadata.annotations.ibm-license-key-applied from Secret cpd-instance1-operator/postgresql-operator-controller-manager-config: ibm-license-key-applied is not found
    E0912 06:57:24.564460    1 reconcile_operand.go:494] Failed to reconcile k8s resource -- Kind: Cluster, NamespacedName: /common-service-db with error: failed to parse path .metadata.annotations.ibm-license-key-applied from Secret cpd-instance1-operator/postgresql-operator-controller-manager-config: ibm-license-key-applied is not found
  

Cause

When you apply a patch for the license key of the embedded EDB PostgreSQL database on an old IBM Cloud Pak foundational services version, the label operator.ibm.com/opreq-control: 'false' is added to the create-postgres-license-config job. This label is preventing Operand Deployment Lifecycle Manager (ODLM) from running the job with the latest image to add an annotation on postgresql-operator-controller-manager-config secret. This annotation is required to successfully create the common-service-db Postgres cluster CR.

Resolving the problem

To resolve the issue, delete the existing create-postgres-license-config job and re-create it. Complete the following steps:

1. Delete the create-postgres-license-config job.

    oc delete job create-postgres-license-config -n <namespace-where-foundational services-operator-are-installed>
   

2. Delete the ODLM pod to re-create the create-postgres-license-config job.

    oc delete pod -n <namespace-where-foundational services-operator-are-installed> -l name=operand-deployment-lifecycle-manager
   

3. Make sure that the create-postgres-license-config job is created and that the job successfully completes.

    oc get job create-postgres-license-config -n <namespace-where-foundational services-operator-are-installed>
   

   You might see the following output:

    NAME                         COMPLETIONS   DURATION   AGE
    create-postgres-license-config   1/1           1s         1m
   

4. Wait for the common-service-db Postgres cluster CR to be successfully created.

    oc get cluster.postgresql.k8s -n <namespace-where-foundational services-are-installed> | grep common-service-db
   

   You might see the following output:

    common-service-db      161d   2           2       Cluster in healthy state                     common-service-db-2
   

   {: pre}