IM operand pods issue during MongoDB to PostgreSQL migration

The Identity Management (IM) operand pods such as platform-auth-service, platform-identity-management, and platform-identity-provider are not created during data migration.

Symptoms

IM operand pods (platform-auth-service, platform-identity-management, and platform-identity-provider) are not created when the IM operator pod is in running status. To check the status of the IM operator pods, run the following command:

   $ oc get authentication.operator.ibm.com example-authentication -oyaml -n <your-foundational-services-namespace>| yq '.metadata.annotations'
   authentication.operator.ibm.com/db-schema-version: 1.0.0
   authentication.operator.ibm.com/migration-complete: "true"
   authentication.operator.ibm.com/retain-migration-artifacts: "true"

Replace <your-foundational-services-namespace> with the namespace where you deployed the foundational services.

Diagonising the problem

To diagonise the IM operand pods issue, complete the following steps:

1. Ensure that common-service-postgresql is included in the ibm-iam-request OperandRequest. If common-service-postgresql is not included in the ibm-iam-request OperandRequest, the issue is in postgresql request from IM operator.

   oc get operandrequest ibm-iam-request -oyaml |yq '.spec.requests[].operands'
   - name: ibm-idp-config-ui-operator
   - bindings:
       protected-im-db:
         configmap: im-datastore-edb-cm
         secret: im-datastore-edb-secret
     name: common-service-postgresql
   

2. Ensure that the cluster custom resource is created. If the custom resource is not created, check operand-deployment-lifecycle-manager pod logs.

   $ oc get cluster
   

   NAME                     AGE   INSTANCES   READY   STATUS                     PRIMARY
   common-service-db        23h   1           1       Cluster in healthy state   common-service-db-1
   

3. Ensure that the common-service-db pods are running. If the pods are not running, check for the StorageClass related issues.

   $ oc get pods |grep common-service-db
   

   common-service-db-1           1/1     Running     0         23h
   

4. Ensure that the im-datastore-edb-cm configmap and im-datastore-edb-secret Secret are created.

   oc get cm -n <your-foundational-services-namespace> | grep im-datastore-edb-cm
   oc get secret  -n <your-foundational-services-namespace> | grep im-datastore-edb-secret
   

   Replace <your-foundational-services-namespace> with the namespace where you deployed the foundational services.

5. Ensure that the MongoDB to PostgreSQL database migration is completed successfully.

   $ oc get authentication.operator.ibm.com example-authentication -oyaml | yq '.metadata.annotations'
   authentication.operator.ibm.com/db-schema-version: 1.0.0
   authentication.operator.ibm.com/migration-complete: "true"
   authentication.operator.ibm.com/retain-migration-artifacts: "true"
   

   If authentication.operator.ibm.com/migration-complete is set to "true", the database migration is successful. If authentication.operator.ibm.com/migration-complete is set to "false", see MongoDB to PosgreSQL database migration fails because of the data issues in MongoDB.

   Note: For the Isolated migration of the cluster with two or more Cloud Paks in the same namespace, the MongoDB data is migrated successfully when you upgrade the first Cloud Pak. To migrate the MongoDB data from other Cloud Paks, run the following command:

      #!/bin/bash
   
      DB_POD="icp-mongodb-0"
   
      # Execute MongoDB rollback commands
      echo 'use samlDB
      db.saml.updateMany({}, {$unset:{migrated: null}})
   
      use platform-db
      db.cloudpak_ibmid_v3.updateMany({}, {$unset:{migrated: null}})
      db.cloudpak_ibmid_v2.updateMany({}, {$unset:{migrated: null}})
      db.Directory.updateMany({}, {$unset:{migrated: null}})
      db.Users.updateMany({}, {$unset:{migrated: null}})
      db.UserPreferences.updateMany({}, {$unset:{migrated: null}})
      db.ZenInstance.updateMany({}, {$unset:{migrated: null}})
      db.ZenInstanceUsers.updateMany({}, {$unset:{migrated: null}})
      db.ScimAttributes.updateMany({}, {$unset:{migrated: null}})
      db.ScimAttributeMapping.updateMany({}, {$unset:{migrated: null}})
      db.Groups.updateMany({}, {$unset:{migrated: null}})
      db.ScimServerUsers.updateMany({}, {$unset:{migrated: null}})
      db.ScimServerGroups.updateMany({}, {$unset:{migrated: null}})
   
      use OAuthDBSchema
      db.OauthClient.updateMany({}, {$unset:{migrated: null}})' | oc exec -ti $DB_POD -- bash -ec 'mongo --host rs0/mongodb:27017 --username $ADMIN_USER --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem'