Upgrading IBM Cloud Pak for Data from Version 3.5

When you upgrade IBM® Cloud Pak for Data, you update the IBM Cloud Pak® for Data platform operator and the IBM Cloud Pak foundational services operator to watch the project where IBM Cloud Pak for Data Version 3.5 is installed. Then, you create a custom resource to install Refresh 9 of Version 4.0 in that project.

Permissions you need for this task
You must be either:
  • A cluster administrator
  • An administrator of the following projects:
    The IBM Cloud Pak foundational services project
    Identified by the ${PROJECT_CPFS_OPS} environment variable.

    Typically the ibm-common-services project.

    The IBM Cloud Pak for Data platform operator project
    Identified by the ${PROJECT_CPD_OPS} environment variable.

    This project can be a separate project, such as cpd-operators or the same project as the IBM Cloud Pak foundational services project.

    The project where Cloud Pak for Data Version 3.5 is installed
    Identified by the ${PROJECT_CPD_INSTANCE} environment variable.
When you need to complete this task
You must complete this task to upgrade each instance of Cloud Pak for Data Version 3.5 on your cluster. For example, if you have two instances of Cloud Pak for Data Version 3.5 on your cluster, you must complete this task twice.
Information you need to confirm before you start this task
Before you upgrade Cloud Pak for Data, confirm:
  • The name of the project where Cloud Pak for Data is installed.
  • The storage classes that you are using for your existing Cloud Pak for Data installation.
    Important: You cannot change the storage classes that are associated with this installation. If you try to change the storage, the upgrade will fail.
    Run the following commands to check which storage classes are in use:
    • To get the primary storage class, run:
      oc get pvc user-home-pvc -o jsonpath='{.spec.storageClassName}'
    • To get the metadata storage class, run:
      oc get pvc datadir-zen-metastoredb-0 -o jsonpath='{.spec.storageClassName}'
Information you need to complete this task
  • The Cloud Pak for Data control plane needs only the restricted security context constraint (SCC).
  • The Cloud Pak for Data control plane uses the following storage classes. If you don't use these storage classes on your cluster, ensure that you have a storage class with an equivalent definition:
    • OpenShift® Container Storage: ocs-storagecluster-cephfs
    • IBM Spectrum® Scale Container Native: ibm-spectrum-scale-sc
    • NFS: managed-nfs-storage
    • Portworx: portworx-shared-gp3
    • IBM Cloud File Storage: ibmc-file-gold-gid or ibm-file-custom-gold-gid

Environments where Data Virtualization is installed

If you have Data Virtualization on Cloud Pak for Data Version 3.5, you must:
  • Back up the list of users who have access to the Data Virtualization service.
  • Copy any JAR files from the JDBC drivers that you downloaded when you configured data source connections in the Data Virtualization service.
For details, see Export users and custom JAR files.
Attention: If you upgrade Cloud Pak for Data without backing up the list of Data Virtualization users, the users will lose access to the Data Virtualization service.

Before you begin

Best practice: You can run the commands in this task exactly as written if you set up environment variables for your installation. For instructions, see Best practice: Setting up install variables.

Ensure that you run the environment variable script before you run the commands in this task.

Ensure that a cluster administrator completed the required upgrade preparation tasks for your environment. Specifically, verify that a cluster administrator completed the following tasks:

  1. If you are using the specialized installation architecture (where the IBM Cloud Pak foundational services operators and Cloud Pak for Data operators are in separate OpenShift projects), ensure that IBM Cloud Pak foundational services is installed. For details, see Installing or upgrading IBM Cloud Pak foundational services.
  2. For environments that use a private container registry, such as air-gapped environments, the Cloud Pak for Data software images are mirrored to the private container registry. For details, see Mirroring images to your private container registry before upgrading from Version 3.5.
  3. The cluster is configured to pull the software images. For details, see Configuring your cluster to pull software images before upgrading from Version 3.5.
  4. The Cloud Pak for Data catalog source exists. For details, see Creating catalog sources.
  5. The Cloud Pak for Data operator subscription and the IBM NamespaceScope Operator subscription exist. For details, see Creating operator subscriptions before upgrading from Version 3.5.

If you do not complete these steps, the Cloud Pak for Data upgrade will fail.

Procedure

To upgrade Cloud Pak for Data:

  1. Log in to the Red Hat® OpenShift Container Platform as a user with sufficient permissions to complete the task:
    oc login ${OCP_URL}
  2. Enable the IBM Cloud Pak for Data platform operator and the IBM Cloud Pak foundational services operator to watch the project where IBM Cloud Pak for Data is installed:
    Express installation (all operators are in the same project)

    Create an operand request to grant permission to the IBM Cloud Pak for Data platform operator and the IBM Cloud Pak foundational services operator to manage the project where Cloud Pak for Data is installed.

    cat <<EOF |oc apply -f -
    apiVersion: operator.ibm.com/v1alpha1
    kind: OperandRequest
    metadata:
      name: empty-request
      namespace: ${PROJECT_CPD_INSTANCE}
    spec:
      requests: []
    EOF

    Specialized installation (IBM Cloud Pak foundational services operators and Cloud Pak for Data operators are in separate projects)

    Update the NamespaceScope object in the Cloud Pak for Data operators project to watch the project where Cloud Pak for Data is installed.

    1. Check whether there is already a NamespaceScope object named cpd-operators in the {PROJECT_CPD_OPS} project:
      oc get NamespaceScope -n ${PROJECT_CPD_OPS}
    2. Complete the appropriate step for your environment:
      The command returns No resources found in the ... namespace
      Run the following command to create the NamespaceScope object:
      cat <<EOF |oc apply -f -
      apiVersion: operator.ibm.com/v1
      kind: NamespaceScope
      metadata:
        name: cpd-operators
        namespace: ${PROJECT_CPD_OPS}
      spec:
        namespaceMembers:
        - ${PROJECT_CPD_OPS}
        - ${PROJECT_CPD_INSTANCE}
      EOF
      The command returns information about the NamespaceScope object
      1. Run the following command to update the NamespaceScope object in the {PROJECT_CPD_OPS} project:
        oc patch NamespaceScope cpd-operators \
        -n ${PROJECT_CPD_OPS} \
        --type json \
        --patch '[{ "op": "add", "path": "/spec/namespaceMembers/0", "value": "${PROJECT_CPD_INSTANCE}" }]'
      2. To confirm that the project was added, run the following command to see the contents of the NamespaceScope YAML:
        oc get NamespaceScope cpd-operators -n ${PROJECT_CPD_OPS} -o yaml
        The namespaceMembers list should contain:
        • The project where the Cloud Pak for Data operators are installed (as specified by ${PROJECT_CPD_OPS}).
        • The project where Cloud Pak for Data is installed (as specified by ${PROJECT_CPD_INSTANCE}).

  3. Create a custom resource to upgrade Cloud Pak for Data. Follow the appropriate guidance for your environment.
    Important: By creating an Ibmcpd custom resource with spec.license.accept: true, you are accepting the license terms for Cloud Pak for Data. You can find links to the relevant licenses in Cloud Pak for Data License Information.

    The cluster uses the recommended storage class names on Red Hat OpenShift Container Storage

    Create a custom resource with the following format.

    Restriction: You can use this format only if your current installation uses the following storage classes:
    storageClass: ocs-storagecluster-cephfs
    zenCoreMetadbStorageClass: ocs-storagecluster-ceph-rbd

    If your installation uses different storage classes, follow the guidance under The cluster uses custom storage classes.

    cat <<EOF |oc apply -f -
    apiVersion: cpd.ibm.com/v1
    kind: Ibmcpd
    metadata:
      name: ibmcpd-cr     # This is the recommended name, but you can change it
      namespace: ${PROJECT_CPD_INSTANCE}
    spec:
      license:
        accept: true
        license: ${LICENSE_CPD}
      storageVendor: ocs
    EOF

    The cluster uses the recommended storage class names on Portworx

    The recommended storage classes are described in Setting up shared persistent storage.

    Create a custom resource with the following format.

    Restriction: You can use this format only if your current installation uses the following storage classes:
    storageClass: portworx-shared-gp3
    zenCoreMetadbStorageClass: portworx-metastoredb-sc

    If your installation uses different storage classes, follow the guidance under The cluster uses custom storage classes.

    cat <<EOF |oc apply -f -
    apiVersion: cpd.ibm.com/v1
    kind: Ibmcpd
    metadata:
      name: ibmcpd-cr     # This is the recommended name, but you can change it
      namespace: ${PROJECT_CPD_INSTANCE}
    spec:
      license:
        accept: true
        license: ${LICENSE_CPD}
      storageVendor: portworx
    EOF

    The cluster uses NFS storage

    The recommended storage class is described in Setting up shared persistent storage.

    Important: You cannot run this command as written. You must edit this command to specify a valid storage classes.
    cat <<EOF |oc apply -f -
    apiVersion: cpd.ibm.com/v1
    kind: Ibmcpd
    metadata:
      name: ibmcpd-cr      # This is the recommended name, but you can change it
      namespace: ${PROJECT_CPD_INSTANCE}
    spec:
      license:
        accept: true
        license: ${LICENSE_CPD}
      storageClass: RWX-storage-class     # Replace with the NFS RWX storage class used by your current installation
    EOF

    The cluster uses custom storage classes

    If your cluster uses storage class names other than those described in Setting up shared persistent storage, you must tell Cloud Pak for Data what storage class names to use.

    Important: You cannot run this command as written. You must edit this command to specify valid storage classes.
    cat <<EOF |oc apply -f -
    apiVersion: cpd.ibm.com/v1
    kind: Ibmcpd
    metadata:
      name: ibmcpd-cr     # This is the recommended name, but you can change it
      namespace: ${PROJECT_CPD_INSTANCE}
    spec:
      version: 4.4.4
      license:
        accept: true
        license: ${LICENSE_CPD}
      storageClass: RWX-storage-class     # Replace with the RWX storage class used by your current installation
      zenCoreMetadbStorageClass: RWO-storage-class     # See "Configuring metadata storage" for details
    EOF
    Important:
    Configuring metadata storage
    Specify the storage class that is returned by the following command:
    oc get pvc datadir-zen-metastoredb-0 -o jsonpath='{.spec.storageClassName}'

    If your current installation does not include a value for this setting, remove or comment out this line in the Ibmcpd custom resource.


Verifying the upgrade

When you create the Ibmcpd custom resource, the IBM Cloud Pak for Data platform operator processes the contents of the custom resource and starts up the microservices that comprise Cloud Pak for Data, including the Cloud Pak for Data control plane.

The Cloud Pak for Data control plane is defined by the ZenSerivce lite-cr custom resource.

To check the status of the upgrade:

  1. Change to the project where Cloud Pak for Data is installed. For example:
    oc project ${PROJECT_CPD_INSTANCE}
  2. Get the status of the control plane (lite-cr):
    oc get ZenService lite-cr -o jsonpath="{.status.zenStatus}{'\n'}"

    The Cloud Pak for Data control plane is ready when the command returns Completed.

    It can take up to 90 minutes for the command to return Completed. If the command still has not returned Completed after 90 minutes, contact IBM Software Support.

  3. Get the URL of the Cloud Pak for Data web client:
    oc get ZenService lite-cr -o jsonpath="{.status.url}{'\n'}"
    The URL has the following format:
    https://cpd-namespace.apps.OCP-default-domain
  4. Get the initial password for the admin user:
    oc extract secret/admin-user-details --keys=initial_admin_password --to=-
    Important: Save the output of this command so that you can log in to the web client. It is strongly recommended that you change the initial password the first time that you log in to the web client.

Choosing an upgrade plan for the Cloud Pak for Data control plane

For Refresh 9 of Version 4.0, the minimum required version of the control plane is 4.4.4, which requires IBM Cloud Pak foundational services Version 3.18.0 or later.

If you install a later version of IBM Cloud Pak foundational services, you might see a later version of the control plane on your cluster depending on the upgrade plan that you choose.

Controlled upgrade (recommended)
It is strongly recommended that you pin the installation to a specific version to prevent unexpected upgrades of the Cloud Pak for Data control plane.
Important: If you have multiple instances of Cloud Pak for Data on your cluster, you must complete the following steps for each instance of Cloud Pak for Data.

To change to the manual upgrade plan:

  1. Update the ZenService custom resource.

    For example, to pin the installation at 4.4.4, run the following command:

    oc patch ZenService lite-cr \
    --namespace ${PROJECT_CPD_INSTANCE} \
    --type=merge \
    --patch '{"spec": {"version":"4.4.4"}}'

    For a list of operand versions supported by the Zen operator, see Cloud Pak for Data operator and operand versions.

Automatic upgrade
By default, the Cloud Pak for Data control plane will be automatically upgraded when you install a newer version of IBM Cloud Pak foundational services on the cluster.
Restriction: Allowing automatic updates is not recommended in production environments where predictability and stability are important. This option is recommended only for short-term installations, such as proof-of-concept deployments.

If you want to continue using the automatic upgrade plan, no additional action is required.

What to do next

After you upgrade the Cloud Pak for Data platform, you must complete the following tasks:

  1. Determine whether you need to update your user management configuration:
    • LDAP integration provided by Cloud Pak for Data

      Cloud Pak for Data provides two options for syncing with your LDAP server:

      Sync on log in (default)
      This is the default method. When this method is used, the platform syncs each user's data when the user logs in to Cloud Pak for Data:
      • The first time that a user logs in to Cloud Pak for Data, the platform creates a user profile and assigns the user the correct user groups based on their LDAP group membership.
      • If the user has logged in before, the platform updates the use group membership based on their LDAP group membership.

      If you want to continue using this method, no additional action is required.

      Periodic sync job
      This option is required if you use nested groups in Microsoft Active Directory. However, this method can cause a lot of overhead for Cloud Pak for Data instances that have large LDAP groups.
      If you want to use this method:
      1. Disable the sync on log in (LDAP_SYNC_ON_LOGIN) by running the following command:
        oc patch configmap product-configmap \
        --namespace ${PROJECT_CPD_INSTANCE} \
        --patch '{"data": {"LDAP_SYNC_ON_LOGIN" : "false"}}'
      2. Delete the usermgmt pods:
        oc delete pod \
        --namespace ${PROJECT_CPD_INSTANCE} \
        -l component=usermgmt
      3. Enable the periodic sync job:
        oc patch cj usermgmt-ldap-sync-cron-job \
        --namespace ${PROJECT_CPD_INSTANCE} \
        --patch '{"spec": {"suspend": false}}'
    • LDAP integration provided by the IAM Service

      If you integrated with the IBM Cloud Pak foundational services Identity and Access Management Service (IAM Service) in Cloud Pak for Data Version 3.5, you must complete Integrating with the IAM Service to re-enable the integration with the IAM Service.

  2. Determine whether you should complete any of the Post-installation tasks.

    For example, if you want to use the manual install plan for operators on your cluster, follow the guidance in Specifying the install plan for operators that are automatically installed by Operand Deployment Lifecycle Manager.

  3. Upgrade the services on your cluster. For details, see Services.