Isolated migration

Isolated migration will first isolate the shared 3.x version of foundational services on the cluster if needed, then fresh install a new instance of foundational services version 4.x. Any data from the original foundational services will be copied to the new instance.

This method can be used if the foundational services you are migrating is shared for the entire cluster or dedicated to a IBM Cloud Pak.

Note: You can choose to migrate by either using a script or using manual steps.

Prerequisite

• See supported starting versions in Migrating foundational services v3.x to 4.x.
• Download scripts and go to the installer_scripts directory. For more information, see Downloading scripts for additional configuration from specific version CASE bundle.
• Set the required cluster permissions to run the isolate.sh and setup_singleton.sh scripts. For more information, see Additional cluster permissions for running scripts .
• Create new CatalogSource or update existing CatalogSource to the new 4.x version.
  • For online migration, reference the required CatalogSource for the corresponding version.
  • For offline migration, reference the upgrade steps to mirror images, and apply the new CatalogSource.

Migrate using a script

Step 1: Isolate and migrate

1. Isolate the original instance of foundational services by running the isolate.sh script.

   Run ./isolate.sh --help for usage instructions. For example, CloudPak instances in cpns1 and cpns2 sharing the same foundational services instance in ibm-common-services, create zenservice in both cpns1, cpns2, and zen-tenant namespace. When you want to isolate cpns1 and upgrade later, run the following script:

   ./isolate.sh --original-cs-ns ibm-common-services --control-ns cs-control-ns --excluded-ns cpns1,zen-tenant
   

2. Migrate cluster singletons by running the ./cp3pt0-deployment/setup_singleton.sh script.

   Run ./cp3pt0-deployment/setup_singleton.sh --help for usage instructions. For example, foundational services is installed in the ibm-common-services namespace and user-created ibm-cert-manager-catalog catalogsource, and ibm-licensing-catalog in the openshift-marketplace namespace. Run the following script:

   ./cp3pt0-deployment/setup_singleton.sh --operator-namespace ibm-common-services --enable-licensing --cert-manager-source ibm-cert-manager-catalog --licensing-source ibm-licensing-catalog --license-accept -v 1
   

   If you have deployed License Service Reporter, use the following command to migrate the IBM License Service Reporter:

   ./cp3pt0-deployment/setup_singleton.sh --operator-namespace ibm-common-services --enable-licensing --enable-license-service-reporter --cert-manager-source ibm-cert-manager-catalog --licensing-source ibm-licensing-catalog --license-accept -v 1
   

3. Define the foundational services version 4.x topology by running the ./cp3pt0-deployment/common/authorize-namespace.sh script.

   Note: You must create the operator namespace and the services namespace for the new topology before running authorize-namespace.sh.

Step 2: Preload data

This step will copy data in the isolated original foundational services version 3.x scope and preload it into the foundational services version 4.x tenant's services namespace, and will be ready to be used by the new foundational services version 4.x services whenever they are installed. Such data includes, but not limited to:

1. IM admin login credentials
2. Mongo data containing IM users

Run the preload_data.sh script. Make sure to pass in the original common services namespace from the original tenant and the services namespace from the new tenant as parameters.

For example: ./preload_data.sh --original-cs-ns <original cs ns> --services-ns <new services ns>. For more details, use the -h option in the script.

See the following notes:

• Security note: Due to the limitations of the community provided mongodump and mongorestore backup and restore tools for MongoDB, during the duration of the preload_data.sh script's runtime, there are non-SSL connections made to the MongoDB instance targeted by the script on Power and Z clusters. During this time, the running MongoDB instance is changed from requireSSL to preferSSL to facilitate the migration of MongoDB data. Once the script has successfully completed, the MongoDB instance is reverted back to requireSSL and all further connections are required to be SSL.
• If new data is introduced after the data has already been copied and before the IBM Cloud Pak has been upgraded, the upgraded IBM Cloud Pak will not have this newly introduced data. If you do something to introduce data while this upgrade process is in progress, the newly introduced data may not be reflected after IBM Cloud Pak upgrade.

Performing the isolated upgrade during a maintenance window is recommended to reduce the risk of this new data loss.

Troubleshooting: If the preload_data.sh script fails to complete due to an error in the middle of the run, complete one of the following actions:

• If an immediate rerun is required, use the same parameters from the original run and add --rerun. This cleans up the leftover resources in the specified services namespace and restarts the preload_data.sh script process immediately.

• If an immediate rerun is not required, use the same parameters from the original run and add --cleanup. This only cleans up the resources in the specified services namespace.

  Check the value for --services-ns because both --rerun and --cleanup remove the existing Mongo DB deployments in this namespace. For more information, check the preload_data.sh script by running the ./preload_data.sh --help command.

Step 3: Fresh install foundational services version 4.x

1. Decide whether you want a simple topology or advanced topology for the foundational services version 4.x tenant.
2. If you want a simple topology, install or upgrade common-service-operator v4 into the one and only namespace for your foundational services version 4.x tenant.

3. If you want an advanced topology, running ./cp3pt0-deployment/setup_tenant.sh script.

   ./cp3pt0-deployment/setup_tenant.sh --operator-namespace <foundational-services-v4-operators-ns> --services-namespace <foundational-services-v4-operands-ns-if-needed> --tethered-namespace <to-be-upgrade-Cloud-Pak-ns> --license-accept
   

4. Verify that common-service-operator is Running.

5. Verify that common-service-maps has a new tenant with namespaces for foundational services version 4.x and the to-be-upgraded IBM Cloud Pak.

Step 4: Upgrade IBM Cloud Pak

At this point, the cluster state is as follows:

• Isolated IBM Cloud Pak and foundational services workloads running
• New foundational services version 4.x tenant installed with just common-service-operator, ODLM, and NSS, and any of the to-be-upgraded IBM Cloud Pak services

• No IM or other foundational services installed yet

• Change channel of IBM Cloud Pak operator(s).

• Verify new or updated OperandRequests exist, for example, requesting ibm-im-operator, ibm-platformui-operator, and so on.
  • IBM Cloud Pak services are responsible for ensuring OperandRequests are updated
• If there is ZenService CR in IBM Cloud Pak namespace(s), remove the csNamespace field if it exists.
• Verify that IBM Cloud Pak works.

If the upgraded IBM Cloud Pak uses SAML or OIDC for single sign-on or if you have configured this, it must be re-registered and the new IM instance will need to be re-configured since the cp-console route and other foundational services routes would have changed due to this instance of foundational services being a fresh installation. For more information, see Migrating identity management.

Migrate using manual steps

Step 1: Isolate original instance of foundational services

Step 1 involves isolating the original foundational services so that it will not reconcile, manage, or interfere with the foundational services version 4.x workloads on the cluster. If there are already dedicated foundational services instances, this step is unnecessary.

Important: Only perform this step if the cluster is using a shared instance of foundational services. The cluster is using a shared instance of foundational services if the common-service-maps configmap in kube-public namespace does not have controlNamespace specified.

1. Edit the CSV of common-service-operator and ODLM and set replicas: 0 to scale them down:

   oc scale deployment -n <namespace> ibm-common-service-operator --replicas=0
   oc scale deployment -n <namespace> operand-deployment-lifecycle-manager --replicas=0
   

2. Create the configmap, common-service-maps, in kube-public namespace if it does not exist already.

   • Add namespaces of all running IBM Cloud Paks on the cluster to common-service-maps as one foundational services version 3.x tenant.
   • Exclude the IBM Cloud Pak namespaces which will be upgraded to foundational services version 4.x.
   • Add namespace of where the shared instance of foundational services is running to the same foundational services version 3.x tenant.

   Example common-service-maps to isolate original instance of foundational services:

      kind: ConfigMap
      apiVersion: v1
      metadata:
          name: common-service-maps
          namespace: kube-public
      data:
        common-service-maps.yaml: |
          controlNamespace: cs-control
          namespaceMapping:
          - requested-from-namespace:
            - cp1-ns1
            - cp2-ns1
            - cp2-ns2
            - ibm-common-services
            map-to-common-service-namespace: ibm-common-services
   

3. Edit the ODLM subscription and change the ISOLATED_MODE: 'false' to ISOLATED_MODE: 'true'

   Note: yq version 4 or later is required.

   oc get subscription.operators.coreos.com operand-deployment-lifecycle-manager-app -n <namespace> -o yaml > sub.yaml
   yq e '.spec.config.env |= (map(select(.name == "ISOLATED_MODE").value |= "true") + [{"name": "ISOLATED_MODE", "value": "true"}] | unique_by(.name))' sub.yaml -i
   oc apply -f sub.yaml
   

4. Delete OperandRegistry and OperandConfig, both called common-service, in the foundational services namespace:

   oc delete operandregistry -n <namespace> --ignore-not-found common-service
   oc delete operandconfig -n <namespace> --ignore-not-found common-service
   

5. Delete all NSS CRs from the namespace where ODLM is running

   Tip: If the oc command is stuck here, remove the finalizer in the NSS CRs.

   oc delete namespacescopes.operator.ibm.com nss-managedby-odlm odlm-scope-managedby-odlm nss-odlm-scope common-service -n <namespace>
   

The tenant in the common-service-maps refers to one array element under the namespaceMapping field as shown in the preceding example.

Step 2: Migrate cluster singletons

Step 2 is necessary because a dedicated instance of foundational services version 3.x requires that singletons be installed in controlNamespace, hence they must be migrated (uninstalled and reinstalled) from ibm-common-services to controlNamespace.

1. Create the namespace specified in controlNamespace if it does not exist already.

   oc create namespace cs-control
   

2. Create configmap ibm-cpp-config in controlNamespace and add deployCSCertManagerOperand: "false" to the data.

3. Delete CertManager CR.

   oc delete certmanagers.operator.ibm.com default
   

4. Delete the ibm-cert-manager-operator subscription and CSV.

   oc delete subscription.operators.coreos.com ibm-cert-manager-operator -n <namespace>
   export CSV=$(oc get csv -n <namespace> | grep ibm-cert-manager | awk '{print $1}')
   oc delete ClusterServiceVersion $CSV -n <namespace>
   

5. To migrate License Service Reporter, complete the following steps:

   1. Decouple the License Service Reporter Persistent Volume (PV) and Persistant Volume Claim (PVC).

      1. Check the status of the License Service Reporter PVC.

         oc get pvc license-service-reporter-pvc --ignore-not-found -n ibm-common-services  --no-headers | awk '{print $2}'
         

         As a reply, you get the PVC status, for example Bound.

      2. If the PVC is in Bound status, get the name of the License Service Reporter PV.

         VOL=$(oc get pvc license-service-reporter-pvc --ignore-not-found -n $ibm-common-services  -o=jsonpath='{.spec.volumeName}')
         

      3. Change the PV's persistentVolumeReclaimPolicy to Retain and label the License Service Reporter PV as LSR PV to enable further License Service Reporter upgrade.

         oc patch pv $VOL -p '{"spec": { "persistentVolumeReclaimPolicy" : "Retain" }}'
         
         oc label pv $VOL license-service-reporter-pv=true --overwrite
         

   2. Delete the IBMLicenseServiceReporter instance.

      oc delete IBMLicenseServiceReporter instance -n ibm-common-services
      

   3. Migrate License Service Reporter to a new namespace.

      1. Create a new namespace for License Service Reporter.

         oc create namespace ${LSR_NAMESPACE}
         

      2. Get the StorageClass from the existing License Service Reporter PV.

         Note: You do not need to retrieve the StorageClass on IBM ROKS.

         LSR_PV_NAME=$(oc get pv -l license-service-reporter-pv=true -o=jsonpath='{.items[0].metadata.name}')
         
         LSR_STORAGE_CLASS=$(oc get pv -l license-service-reporter-pv=true -o=jsonpath='{.items[0].spec.storageClassName}')
         

      3. Create a new PVC with the existing PV in the new License Service Reporter namespace.

         apiVersion: v1
         kind: PersistentVolumeClaim
         metadata:
            name: license-service-reporter-pvc
            namespace: ${LSR_NAMESPACE}
         spec:
            accessModes:
            - ReadWriteOnce
            resources:
               requests:
                     storage: 1Gi
            storageClassName: "${LSR_STORAGE_CLASS}"
            volumeMode: Filesystem
            volumeName: ${LSR_PV_NAME}
         

   4. Reinstall License Service Reporter operator in the new namespace. For more information, see Installing License Service Reporter from online repository.

6. Delete licensing CR.

   oc delete ibmlicensing instance
   

7. Delete ibm-license-operator Subscription and CSV.

   oc delete subscription.operators.coreos.com ibm-licensing-operator -n <namespace>
   export CSV=$(oc get csv -n <namespace> | grep ibm-licensing | awk '{print $1}')
   oc delete ClusterServiceVersion $CSV -n <namespace>
   

8. Copy licensing data to controlNamespace.

   The following is a list of possible licensing configmaps to copy:

   ibm-licensing-config
   ibm-licensing-annotations
   ibm-licensing-products
   ibm-licensing-products-vpc-hour
   ibm-licensing-cloudpaks
   ibm-licensing-products-groups
   ibm-licensing-cloudpaks-groups
   ibm-licensing-cloudpaks-metrics
   ibm-licensing-products-metrics
   ibm-licensing-products-metrics-groups
   ibm-licensing-cloudpaks-metrics-groups
   ibm-licensing-services
   

9. Delete the two Crossplane Subscriptions and CR, if they exist.

   oc get configuration.pkg.ibm.crossplane.io -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found configuration.pkg.ibm.crossplane.io
   oc get lock.pkg.ibm.crossplane.io -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found lock.pkg.ibm.crossplane.io
   oc get ProviderConfig -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found ProviderConfig
   
   oc delete subscription.operators.coreos.com ibm-crossplane-provider-kubernetes-operator-app -n <control-namespace>
   oc delete subscription.operators.coreos.com ibm-crossplane-provider-ibm-cloud-operator-app -n <control-namespace>
   oc delete subscription.operators.coreos.com ibm-crossplane-operator-app -n <control-namespace>
   

10. Delete the Crossplane CSV.

    export CSV=$(oc get csv -n <namespace> | grep ibm-crossplane-provider | awk '{print $1}')
    oc delete ClusterServiceVersion $CSV -n <namespace>
    
    export CSV=$(oc get csv -n <namespace> | grep ibm-crossplane-operator | awk '{print $1}')
    oc delete ClusterServiceVersion $CSV -n <namespace>
    

11. Verify that ibm-cert-manager-operator, ibm-licensing-operator, and crossplane operators are all installed in controlNamespace.

    • Note: Licensing and crossplane only apply if they existed before deletion.

12. Install global cert-manager (ibm-cert-manager-operator v4 or Red Hat cert-manager). See Installing IBM Cert Manager by Console or see Red Hat's instructions.

13. Update the channel of ibm-licensing-operator Subscription to v4.2 or reinstall ibm-licensing-operator v4.2 in target namespace.

14. Scale back up the common-service-operator and ODLM.

    oc scale deployment -n <namespace> ibm-common-service-operator --replicas=1
    oc scale deployment -n <namespace> operand-deployment-lifecycle-manager -- replicas=1
    

Example ibm-cpp-config:

kind: ConfigMap
apiVersion: v1
metadata:
 name: ibm-cpp-config
 namespace: ibm-common-services
data:
 deployCSCertManagerOperands: "false"

Step 3: Defining foundational services version 4.x topology

Important: Do not modify the common-service-maps in this step.

1. Create the namespace for foundational services version 4.x

2. Optional Do not do this step if the IBM Cloud Pak installs everything into one namespace

   a. Create all the namespaces necessary for the foundational services version 4.x workloads and IBM Cloud Pak's workloads

   b. Pre-emptively authorize namespace-scope-operator (not installed yet) to manage all Cloud Pak namespaces from the foundational services namespace by creating the following Role and RoleBinding in each of the namespaces created in step 3.2

For example:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
subjects:
- kind: ServiceAccount
  name: ibm-namespace-scope-operator
  namespace: <namespace_where_namespace-scope-operator_is_installed>
roleRef:
  kind: Role
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
  apiGroup: rbac.authorization.k8s.io

Step 4: Preload data

This step will copy data in the isolated original foundational services version 3.x scope and preload it into the foundational services version 4.x tenant's services namespace, and will be ready to be used by the new foundational services version 4.x services whenever they are installed. Such data includes, but not limited to:

• IM admin login credentials

• Mongo data containing IM users

• Note: when copying resources from one namespace to another, use extra caution to remove instance specific information, such as the namespace value, uid, resourceVersion, ownerReferences, etc.

• Copy the following secrets from the original foundational services namespace and create new secrets with the same name and data in the new namespace created in step 3.1:

  • platform-auth-idp-credentials
  • platform-auth-ldaps-ca-cert

• Copy the following configmaps from the original foundational services namespace and create new configmaps with the same name and data in the new namespace created in step 3.1:

  • ibm-cpp-config
  • common-web-ui-config
  • platform-auth-idp

• Copy mongo data

Note: If new data is introduced after the data has already been copied and before the IBM Cloud Pak has been upgraded, the upgraded IBM Cloud Pak will not have this newly introduced data. If you do something to introduce data while this upgrade process is in progress, the newly introduced data may not be reflected after IBM Cloud Pak upgrade.

Performing the isolated upgrade during a maintenance window is recommended to reduce the risk of this new data loss.

If you encounter issues after running the preload_data.sh script, see preload_data.sh fails due to cert manager.

Step 5: Fresh install foundational services version 4.x

1. Decide whether you want a simple topology or advanced topology for the foundational services version 4.x tenant.
2. If you want a simple topology, install or upgrade common-service-operator v4 into the one and only namespace for your foundational services version 4.x tenant.

3. If you want an advanced topology, running the ./cp3pt0-deployment/setup_tenant.sh script.

   a. Install the ibm-namespace-scope-operator into the namespace created in step 3.1. This will now be known as the operatorNamespace for foundational services.

   b. Install the common-service-operator into the same namespace.

   c. Create a NamespaceScope CR

   apiVersion: operator.ibm.com/v1
   kind: NamespaceScope
   metadata:
      name: common-service
   spec:
      csvInjector:
         enable: true
      namespaceMembers: <comma-delimited list of namespaces from step 3, e.g. cloudpak1, cpfs1>
      restartLabels:
         intent: projected
   

   d. Install the common-service-operator into the operatorNamespace.

   e. Verify that a CommonService CR named common-service exists in this namespace. Configure the default CommonService CR

   apiVersion: operator.ibm.com/v3
   kind: CommonService
   metadata:
      name: common-service
   spec:
      operatorNamespace: operatorNamespace # this should already be set by default, but if not, set it
      servicesNamespace: <namespace where operands will be installed into> # this is optional
      size: <set as either small, medium, or large> # this is optional
   

4. Verify that common-service-operator is Running.

5. Verify that common-service-maps has a new tenant with namespaces for foundational services version 4.x and the to-be-upgraded IBM Cloud Pak.

Step 6: Upgrade IBM Cloud Pak

At this point, the cluster state is as follows:

• Isolated IBM Cloud Pak and foundational services workloads running
• New foundational services version 4.x tenant installed with just common-service-operator, ODLM, and NSS, and any of the to-be-upgraded IBM Cloud Pak services

• No IM or other foundational services installed yet

• Change channel of IBM Cloud Pak operator(s).

• Verify new or updated OperandRequests exist, for example, requesting ibm-im-operator, ibm-platformui-operator, and so on.
  • IBM Cloud Pak services are responsible for ensuring OperandRequests are updated
• If there is ZenService CR in IBM Cloud Pak namespace(s), remove the csNamespace field if it exists.
• Verify that IBM Cloud Pak works.

If the upgraded IBM Cloud Pak uses SAML or OIDC for single sign-on or if you have configured this, it must be re-registered and the new IM instance will need to be re-configured since the cp-console route and other foundational services routes would have changed due to this instance of Bedfoundational services being a fresh installation. For more information, see Migrating identity management.

Note: For the Isolated migration of the cluster with two or more Cloud Paks in the same namespace, the MongoDB data is migrated successfully when you upgrade the first Cloud Pak. To migrate the MongoDB data from other Cloud Paks, run the following command:

   #!/bin/bash

   DB_POD="icp-mongodb-0"

   # Execute MongoDB rollback commands
   echo 'use samlDB
   db.saml.updateMany({}, {$unset:{migrated: null}})

   use platform-db
   db.cloudpak_ibmid_v3.updateMany({}, {$unset:{migrated: null}})
   db.cloudpak_ibmid_v2.updateMany({}, {$unset:{migrated: null}})
   db.Directory.updateMany({}, {$unset:{migrated: null}})
   db.Users.updateMany({}, {$unset:{migrated: null}})
   db.UserPreferences.updateMany({}, {$unset:{migrated: null}})
   db.ZenInstance.updateMany({}, {$unset:{migrated: null}})
   db.ZenInstanceUsers.updateMany({}, {$unset:{migrated: null}})
   db.ScimAttributes.updateMany({}, {$unset:{migrated: null}})
   db.ScimAttributeMapping.updateMany({}, {$unset:{migrated: null}})
   db.Groups.updateMany({}, {$unset:{migrated: null}})
   db.ScimServerUsers.updateMany({}, {$unset:{migrated: null}})
   db.ScimServerGroups.updateMany({}, {$unset:{migrated: null}})

   use OAuthDBSchema
   db.OauthClient.updateMany({}, {$unset:{migrated: null}})' | oc exec -ti $DB_POD -- bash -ec 'mongo --host rs0/mongodb:27017 --username $ADMIN_USER --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem'