Isolated migration

1. Isolate the original instance of foundational services by running the isolate.sh script.

   Run ./isolate.sh --help for usage instructions. For example, CloudPak instances in cpns1 and cpns2 sharing the same foundational services instance in ibm-common-services, create zenservice in both cpns1, cpns2, and zen-tenant namespace. When you want to isolate cpns1 and upgrade later, run the following script:

   ./isolate.sh --original-cs-ns ibm-common-services --control-ns cs-control-ns --excluded-ns cpns1,zen-tenant
    

2. Migrate cluster singletons by running the ./cp3pt0-deployment/setup_singleton.sh script.

   Run ./cp3pt0-deployment/setup_singleton.sh --help for usage instructions. For example, foundational services is installed in the ibm-common-services namespace and user-created ibm-cert-manager-catalog catalogsource, and ibm-licensing-catalog in the openshift-marketplace namespace. Run the following script:

   ./cp3pt0-deployment/setup_singleton.sh --operator-namespace ibm-common-services --enable-licensing --cert-manager-source ibm-cert-manager-catalog --licensing-source ibm-licensing-catalog --license-accept -v 1
    

   If you have deployed License Service Reporter, use the following command to migrate the IBM License Service Reporter:

   ./cp3pt0-deployment/setup_singleton.sh --operator-namespace ibm-common-services --enable-licensing --enable-license-service-reporter --cert-manager-source ibm-cert-manager-catalog --licensing-source ibm-licensing-catalog --license-accept -v 1
    

3. Define the foundational services version 4.x topology by running the ./cp3pt0-deployment/common/authorize-namespace.sh script.

   Note: You must create the operator namespace and the services namespace for the new topology before running authorize-namespace.sh.

Complete the following steps to automatically preload Mongo data by using a script. To preload Mongo data manually, see Preloading Mongo data.

This step will copy data in the isolated original foundational services version 3.x scope and preload it into the foundational services version 4.x tenant's services namespace, and will be ready to be used by the new foundational services version 4.x services whenever they are installed. Such data includes, but not limited to:

1. IM admin login credentials
2. Mongo data containing IM users

Run the preload_data.sh script. Make sure to pass in the original common services namespace from the original tenant and the services namespace from the new tenant as parameters.

For example: ./preload_data.sh --original-cs-ns <original cs ns> --services-ns <new services ns>. For more details, use the -h option in the script.

See the following notes:

• Security note: Due to the limitations of the community provided mongodump and mongorestore backup and restore tools for MongoDB, during the duration of the preload_data.sh script's runtime, there are non-SSL connections made to the MongoDB instance targeted by the script on Power and Z clusters. During this time, the running MongoDB instance is changed from requireSSL to preferSSL to facilitate the migration of MongoDB data. Once the script has successfully completed, the MongoDB instance is reverted back to requireSSL and all further connections are required to be SSL.
• If new data is introduced after the data has already been copied and before the IBM Cloud Pak has been upgraded, the upgraded IBM Cloud Pak will not have this newly introduced data. If you do something to introduce data while this upgrade process is in progress, the newly introduced data may not be reflected after IBM Cloud Pak upgrade.

Performing the isolated upgrade during a maintenance window is recommended to reduce the risk of this new data loss.

Troubleshooting: If the preload_data.sh script fails to complete due to an error in the middle of the run, complete one of the following actions:

• If an immediate rerun is required, use the same parameters from the original run and add --rerun. This cleans up the leftover resources in the specified services namespace and restarts the preload_data.sh script process immediately.
• If an immediate rerun is not required, use the same parameters from the original run and add --cleanup. This only cleans up the resources in the specified services namespace.

Check the value for --services-ns because both --rerun and --cleanup remove the existing Mongo DB deployments in this namespace. For more information, check the preload_data.sh script by running the ./preload_data.sh --help command.

1. Decide whether you want a simple topology or advanced topology for the foundational services version 4.x tenant.
2. If you want a simple topology, install or upgrade common-service-operator v4 into the one and only namespace for your foundational services version 4.x tenant.
3. If you want an advanced topology, running ./cp3pt0-deployment/setup_tenant.sh script.

   ./cp3pt0-deployment/setup_tenant.sh --operator-namespace <foundational-services-v4-operators-ns> --services-namespace <foundational-services-v4-operands-ns-if-needed> --tethered-namespace <to-be-upgrade-Cloud-Pak-ns> --license-accept
    

4. Verify that common-service-operator is Running.
5. Verify that common-service-maps has a new tenant with namespaces for foundational services version 4.x and the to-be-upgraded IBM Cloud Pak.

At this point, the cluster state is as follows:

• Isolated IBM Cloud Pak and foundational services workloads running
• New foundational services version 4.x tenant installed with just common-service-operator, ODLM, and NSS, and any of the to-be-upgraded IBM Cloud Pak services
• No IM or other foundational services installed yet

1. Change channel of IBM Cloud Pak operator(s).
2. Verify new or updated OperandRequests exist, for example, requesting ibm-im-operator, ibm-platformui-operator, and so on.
   • IBM Cloud Pak services are responsible for ensuring OperandRequests are updated
3. If there is ZenService CR in IBM Cloud Pak namespace(s), remove the csNamespace field if it exists.
4. Verify that IBM Cloud Pak works.

If the upgraded IBM Cloud Pak is configured with SAML or OIDC for single sign-on, you need to register SAML or OIDC clients and configure the new IM instance again since the cp-console route and other foundational services routes are changed due to the fresh installation of foundational services. You need to update the SAML IdP URLs (AssertionConsumerService URL and Service Provider (SP) metadata URL) on the IdP side or upload the new metadata to enable OIDC or SAML single sign-on in the upgraded IBM Cloud Pak. For more information, see Migrating identity management.

The default configuration of AssertionConsumerService and SP metadata URLs are https://: /ibm/saml20/defaultSP/acs and https://: /ibm/saml20/defaultSP/samlmetadata. You can update the Liberty SingleLogoutService URL (https://<hostname>:<sslport>/ibm/saml20/<SP configuration ID>/slo) and Liberty SP metadata (https://<hostname>:<sslport>/ibm/saml20/<SP configuration ID>/samlmetadata) to configure the SP for single logout.

The cp-console URL is not configured with the namespace before you upgrade IBM Cloud Pak. If you configure the cp-console URL with the namespace, the existing cp-console routes on the IdP side are also configured with the same namespace after you upgrade IBM Cloud Pak. After upgrade, ensure that you update the IM routes on the IdP side with the same namespace.

Step 1 involves isolating the original foundational services so that it will not reconcile, manage, or interfere with the foundational services version 4.x workloads on the cluster. If there are already dedicated foundational services instances, this step is unnecessary.

1. Edit the CSV of common-service-operator and ODLM and set replicas: 0 to scale them down:

   oc scale deployment -n <namespace> ibm-common-service-operator --replicas=0
   oc scale deployment -n <namespace> operand-deployment-lifecycle-manager --replicas=0
    

2. Create the configmap, common-service-maps, in kube-public namespace if it does not exist already.

   • Add namespaces of all running IBM Cloud Paks on the cluster to common-service-maps as one foundational services version 3.x tenant.
   • Exclude the IBM Cloud Pak namespaces which will be upgraded to foundational services version 4.x.
   • Add namespace of where the shared instance of foundational services is running to the same foundational services version 3.x tenant.

   Example common-service-maps to isolate original instance of foundational services:

   kind: ConfigMap
   apiVersion: v1
   metadata:
       name: common-service-maps
       namespace: kube-public
   data:
     common-service-maps.yaml: |
       controlNamespace: cs-control
       namespaceMapping:
       - requested-from-namespace:
         - cp1-ns1
         - cp2-ns1
         - cp2-ns2
         - ibm-common-services
         map-to-common-service-namespace: ibm-common-services
    

3. Edit the ODLM subscription and change the ISOLATED_MODE: 'false' to ISOLATED_MODE: 'true'

   Note: yq version 4 or later is required.

   oc get subscription.operators.coreos.com operand-deployment-lifecycle-manager-app -n <namespace> -o yaml > sub.yaml
   yq e '.spec.config.env |= (map(select(.name == "ISOLATED_MODE").value |= "true") + [{"name": "ISOLATED_MODE", "value": "true"}] | unique_by(.name))' sub.yaml -i
   oc apply -f sub.yaml
    

4. Delete OperandRegistry and OperandConfig, both called common-service, in the foundational services namespace:

   oc delete operandregistry -n <namespace> --ignore-not-found common-service
   oc delete operandconfig -n <namespace> --ignore-not-found common-service
    

5. Delete all NSS CRs from the namespace where ODLM is running

   Tip: If the oc command is stuck here, remove the finalizer in the NSS CRs.

   oc delete namespacescopes.operator.ibm.com nss-managedby-odlm odlm-scope-managedby-odlm nss-odlm-scope common-service -n <namespace>
    

The tenant in the common-service-maps refers to one array element under the namespaceMapping field as shown in the preceding example.

Step 2 is necessary because a dedicated instance of foundational services version 3.x requires that singletons be installed in controlNamespace, hence they must be migrated (uninstalled and reinstalled) from ibm-common-services to controlNamespace.

1. Create the namespace specified in controlNamespace if it does not exist already.

   oc create namespace cs-control
    

2. Create configmap ibm-cpp-config in controlNamespace and add deployCSCertManagerOperand: "false" to the data.

3. Delete CertManager CR.

   oc delete certmanagers.operator.ibm.com default
    

4. Delete the ibm-cert-manager-operator subscription and CSV.

   oc delete subscription.operators.coreos.com ibm-cert-manager-operator -n <namespace>
   export CSV=$(oc get csv -n <namespace> | grep ibm-cert-manager | awk '{print $1}')
   oc delete ClusterServiceVersion $CSV -n <namespace>
    

5. To migrate License Service Reporter, complete the following steps:

   1. Decouple the License Service Reporter Persistent Volume (PV) and Persistant Volume Claim (PVC).

   2. Check the status of the License Service Reporter PVC.

      oc get pvc license-service-reporter-pvc --ignore-not-found -n ibm-common-services  --no-headers | awk '{print $2}'
       

      As a reply, you get the PVC status, for example Bound.

   3. If the PVC is in Bound status, get the name of the License Service Reporter PV.

      VOL=$(oc get pvc license-service-reporter-pvc --ignore-not-found -n $ibm-common-services  -o=jsonpath='{.spec.volumeName}')
       

   4. Change the PV's persistentVolumeReclaimPolicy to Retain and label the License Service Reporter PV as LSR PV to enable further License Service Reporter upgrade.

      oc patch pv $VOL -p '{"spec": { "persistentVolumeReclaimPolicy" : "Retain" }}'
      
      oc label pv $VOL license-service-reporter-pv=true --overwrite
       

   5. Delete the IBMLicenseServiceReporter instance.

      oc delete IBMLicenseServiceReporter instance -n ibm-common-services
       

   6. Migrate License Service Reporter to a new namespace.

   7. Create a new namespace for License Service Reporter.

      oc create namespace ${LSR_NAMESPACE}
       

   8. Get the StorageClass from the existing License Service Reporter PV.

      Note: You do not need to retrieve the StorageClass on IBM ROKS.

      LSR_PV_NAME=$(oc get pv -l license-service-reporter-pv=true -o=jsonpath='{.items[0].metadata.name}')
      
      LSR_STORAGE_CLASS=$(oc get pv -l license-service-reporter-pv=true -o=jsonpath='{.items[0].spec.storageClassName}')
       

   9. Create a new PVC with the existing PV in the new License Service Reporter namespace.

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
         name: license-service-reporter-pvc
         namespace: ${LSR_NAMESPACE}
      spec:
         accessModes:
         - ReadWriteOnce
         resources:
            requests:
                  storage: 1Gi
         storageClassName: "${LSR_STORAGE_CLASS}"
         volumeMode: Filesystem
         volumeName: ${LSR_PV_NAME}
       

   10. Reinstall License Service Reporter operator in the new namespace. For more information, see Installing License Service Reporter from online repository.

6. Delete licensing CR.

   oc delete ibmlicensing instance
    

7. Delete ibm-license-operator Subscription and CSV.

   oc delete subscription.operators.coreos.com ibm-licensing-operator -n <namespace>
   export CSV=$(oc get csv -n <namespace> | grep ibm-licensing | awk '{print $1}')
   oc delete ClusterServiceVersion $CSV -n <namespace>
    

8. Copy licensing data to controlNamespace.

   The following is a list of possible licensing configmaps to copy:

   ibm-licensing-config
   ibm-licensing-annotations
   ibm-licensing-products
   ibm-licensing-products-vpc-hour
   ibm-licensing-cloudpaks
   ibm-licensing-products-groups
   ibm-licensing-cloudpaks-groups
   ibm-licensing-cloudpaks-metrics
   ibm-licensing-products-metrics
   ibm-licensing-products-metrics-groups
   ibm-licensing-cloudpaks-metrics-groups
   ibm-licensing-services
    

9. Delete the two Crossplane Subscriptions and CR, if they exist.

   oc get configuration.pkg.ibm.crossplane.io -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found configuration.pkg.ibm.crossplane.io
   oc get lock.pkg.ibm.crossplane.io -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found lock.pkg.ibm.crossplane.io
   oc get ProviderConfig -A --no-headers | awk '{print $1}' | xargs ${OC} delete --ignore-not-found ProviderConfig
   
   oc delete subscription.operators.coreos.com ibm-crossplane-provider-kubernetes-operator-app -n <control-namespace>
   oc delete subscription.operators.coreos.com ibm-crossplane-provider-ibm-cloud-operator-app -n <control-namespace>
   oc delete subscription.operators.coreos.com ibm-crossplane-operator-app -n <control-namespace>
    

10. Delete the Crossplane CSV.

    export CSV=$(oc get csv -n <namespace> | grep ibm-crossplane-provider | awk '{print $1}')
    oc delete ClusterServiceVersion $CSV -n <namespace>
    
    export CSV=$(oc get csv -n <namespace> | grep ibm-crossplane-operator | awk '{print $1}')
    oc delete ClusterServiceVersion $CSV -n <namespace>
     

11. Verify that ibm-cert-manager-operator, ibm-licensing-operator, and crossplane operators are all installed in controlNamespace.

    • Licensing and crossplane only apply if they existed before deletion.

12. Install global cert-manager (ibm-cert-manager-operator v4 or Red Hat cert-manager). See Installing IBM Cert Manager by Console or see Red Hat's instructions.

13. Update the channel of ibm-licensing-operator Subscription to v4.2 or reinstall ibm-licensing-operator v4.2 in target namespace.

14. Scale back up the common-service-operator and ODLM.

    oc scale deployment -n <namespace> ibm-common-service-operator --replicas=1
    oc scale deployment -n <namespace> operand-deployment-lifecycle-manager -- replicas=1
     

Example ibm-cpp-config:

kind: ConfigMap
apiVersion: v1
metadata:
 name: ibm-cpp-config
 namespace: ibm-common-services
data:
 deployCSCertManagerOperands: "false"
 

1. Create the namespace for foundational services version 4.x

2. Optional Do not do this step if the IBM Cloud Pak installs everything into one namespace

   a. Create all the namespaces necessary for the foundational services version 4.x workloads and IBM Cloud Pak's workloads

   b. Pre-emptively authorize namespace-scope-operator (not installed yet) to manage all Cloud Pak namespaces from the foundational services namespace by creating the following Role and RoleBinding in each of the namespaces created in step 3.2

For example:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
rules:
- apiGroups:
  - "*"
  resources:
  - "*"
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
  - deletecollection
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
subjects:
- kind: ServiceAccount
  name: ibm-namespace-scope-operator
  namespace: <namespace_where_namespace-scope-operator_is_installed>
roleRef:
  kind: Role
  name: nss-managed-role-from-<namespace_where_namespace-scope-operator_is_installed>
  apiGroup: rbac.authorization.k8s.io
 

This step will copy data in the isolated original foundational services version 3.x scope and preload it into the foundational services version 4.x tenant's services namespace, and will be ready to be used by the new foundational services version 4.x services whenever they are installed. Such data includes, but not limited to:

• IM admin login credentials
• Mongo data containing IM users

• Note: when copying resources from one namespace to another, use extra caution to remove instance specific information, such as the namespace value, uid, resourceVersion, ownerReferences, etc.

1. Copy the following secrets from the original foundational services namespace and create new secrets with the same name and data in the new namespace created in step 3.1:

   • platform-auth-idp-credentials
   • platform-auth-ldaps-ca-cert

2. Copy the following configmaps from the original foundational services namespace and create new configmaps with the same name and data in the new namespace created in step 3.1:

   • ibm-cpp-config
   • common-web-ui-config
   • platform-auth-idp

3. Copy mongo data

Performing the isolated upgrade during a maintenance window is recommended to reduce the risk of this new data loss.

If you encounter issues after running the preload_data.sh script, see preload_data.sh fails due to cert manager.

1. Decide whether you want a simple topology or advanced topology for the foundational services version 4.x tenant.

2. If you want a simple topology, install or upgrade common-service-operator v4 into the one and only namespace for your foundational services version 4.x tenant.

3. If you want an advanced topology, running the ./cp3pt0-deployment/setup_tenant.sh script.

   a. Install the ibm-namespace-scope-operator into the namespace created in step 3.1. This will now be known as the operatorNamespace for foundational services.

   b. Install the common-service-operator into the same namespace.

   c. Create a NamespaceScope CR

   apiVersion: operator.ibm.com/v1
   kind: NamespaceScope
   metadata:
      name: common-service
   spec:
      csvInjector:
         enable: true
      namespaceMembers: <comma-delimited list of namespaces from step 3, e.g. cloudpak1, cpfs1>
      restartLabels:
         intent: projected
    

   d. Install the common-service-operator into the operatorNamespace.

   e. Verify that a CommonService CR named common-service exists in this namespace. Configure the default CommonService CR

   apiVersion: operator.ibm.com/v3
   kind: CommonService
   metadata:
      name: common-service
   spec:
      operatorNamespace: operatorNamespace # this should already be set by default, but if not, set it
      servicesNamespace: <namespace where operands will be installed into> # this is optional
      size: <set as either small, medium, or large> # this is optional
    

4. Verify that common-service-operator is Running.

5. Verify that common-service-maps has a new tenant with namespaces for foundational services version 4.x and the to-be-upgraded IBM Cloud Pak.

At this point, the cluster state is as follows:

• Isolated IBM Cloud Pak and foundational services workloads running
• New foundational services version 4.x tenant installed with just common-service-operator, ODLM, and NSS, and any of the to-be-upgraded IBM Cloud Pak services
• No IM or other foundational services installed yet

1. Change channel of IBM Cloud Pak operator(s).
2. Verify new or updated OperandRequests exist, for example, requesting ibm-im-operator, ibm-platformui-operator, and so on.
   • IBM Cloud Pak services are responsible for ensuring OperandRequests are updated
3. If there is ZenService CR in IBM Cloud Pak namespace(s), remove the csNamespace field if it exists.
4. Verify that IBM Cloud Pak works.

If the upgraded IBM Cloud Pak is configured with SAML or OIDC for single sign-on, you need to register SAML or OIDC clients and configure the new IM instance again since the cp-console route and other foundational services routes are changed due to the fresh installation of foundational services. You need to update the SAML IdP URLs (AssertionConsumerService URL and Service Provider (SP) metadata URL) on the IdP side or upload the new metadata to enable OIDC or SAML single sign-on in the upgraded IBM Cloud Pak. For more information, see Migrating identity management.

The default configuration of AssertionConsumerService and SP metadata URLs are https://: /ibm/saml20/defaultSP/acs and https://: /ibm/saml20/defaultSP/samlmetadata. You can update the Liberty SingleLogoutService URL (https://<hostname>:<sslport>/ibm/saml20/<SP configuration ID>/slo) and Liberty SP metadata (https://<hostname>:<sslport>/ibm/saml20/<SP configuration ID>/samlmetadata) to configure the SP for single logout.

The cp-console URL is not configured with the namespace before you upgrade IBM Cloud Pak. If you configure the cp-console URL with the namespace, the existing cp-console routes on the IdP side are also configured with the same namespace after you upgrade IBM Cloud Pak. After upgrade, ensure that you update the IM routes on the IdP side with the same namespace.

#!/bin/bash

DB_POD="icp-mongodb-0"

# Execute MongoDB rollback commands
echo 'use samlDB
db.saml.updateMany({}, {$unset:{migrated: null}})

use platform-db
db.cloudpak_ibmid_v3.updateMany({}, {$unset:{migrated: null}})
db.cloudpak_ibmid_v2.updateMany({}, {$unset:{migrated: null}})
db.Directory.updateMany({}, {$unset:{migrated: null}})
db.Users.updateMany({}, {$unset:{migrated: null}})
db.UserPreferences.updateMany({}, {$unset:{migrated: null}})
db.ZenInstance.updateMany({}, {$unset:{migrated: null}})
db.ZenInstanceUsers.updateMany({}, {$unset:{migrated: null}})
db.ScimAttributes.updateMany({}, {$unset:{migrated: null}})
db.ScimAttributeMapping.updateMany({}, {$unset:{migrated: null}})
db.Groups.updateMany({}, {$unset:{migrated: null}})
db.ScimServerUsers.updateMany({}, {$unset:{migrated: null}})
db.ScimServerGroups.updateMany({}, {$unset:{migrated: null}})

use OAuthDBSchema
db.OauthClient.updateMany({}, {$unset:{migrated: null}})' | oc exec -ti $DB_POD -- bash -ec 'mongo --host rs0/mongodb:27017 --username $ADMIN_USER --password $ADMIN_PASSWORD --authenticationDatabase admin --ssl --sslCAFile /data/configdb/tls.crt --sslPEMKeyFile /work-dir/mongo.pem'