Upgrading Db2 Big SQL

A project administrator can upgrade the Db2 Big SQL service on IBM® Cloud Pak for Data.

Before you begin

Required role: To complete this task, you must be an administrator of the project (namespace) where Db2 Big SQL is installed.

Before you upgrade Db2 Big SQL, ensure that:

If you are upgrading multiple services on your cluster, you must run the upgrades one at a time and wait until the upgrade completes before upgrading another service. You cannot run the upgrades in parallel.

Preparing for the upgrade

Do the following steps to prepare for the upgrade:

  • After the Cloud Pak for Data control plane upgrade is completed, you must create a cpd-cli profile.

    To upgrade service instances, you must have a cpd-cli profile on your local machine. Ensure that your profile points to the instance of Cloud Pak for Data where the service instances exist. Your profile enables Cloud Pak for Data to ensure that you have the appropriate permissions to upgrade the service instances. For details, see Creating a cpd-cli profile.

  • (Recommended) Mark the Db2 Big SQL Persistent Volume (PV) for retention.

    Mark the Db2 Big SQL PV for retention in case the associated Persistent Volume Claim was deleted.

    oc patch pv $(oc get pv | grep ${NAMESPACE}/bigsql-engine | tr -s ' ' | cut -f 1 -d ' ') --patch '{"spec": { "persistentVolumeReclaimPolicy": "Retain"}}'
  • Scale down the Db2 Big SQL instance to one worker.

    To simplify the upgrade process, scale down the Db2 Big SQL instance to one worker.

    oc scale statefulset worker --replicas=1
  • Prepare and stop the Db2 Big SQL instance.

    Run the Db2 Big SQL upgrade preparation and stop scripts on the Db2 Big SQL head pod.

    1. Get the name of the Db2 Big SQL head pod:
      head_pod=$(oc get pods -l app=db2-bigsql,bigsql-node-role=head --no-headers=true -o=custom-columns=NAME:.metadata.name)
    2. Copy the scripts to the head pod.
      wget https://ibm-open-platform.ibm.com/repos/BigSQL/rhel/7/x86_64/CP4D/scripts/bigsql-upgrade-prep.sh
      chmod +x /tmp/bigsql-upgrade-prep.sh
      oc cp bigsql-upgrade-prep.sh $head_pod:/tmp/bigsql-upgrade-prep.sh
      wget https://ibm-open-platform.ibm.com/repos/BigSQL/rhel/7/x86_64/CP4D/scripts/bigsql-stop.sh
      chmod +x /tmp/bigsql-stop.sh
      oc cp bigsql-stop.sh $head_pod:/tmp/bigsql-stop.sh
    3. Run the upgrade preparation script on the head pod.

      If the instance is connected to a remote object store, an existing bucket name must be passed as a parameter to the script.

      # If connected to an Hadoop cluster
      oc exec -it $head_pod -- bash -lc "/tmp/bigsql-upgrade-prep.sh"
      
      # If connected to an Object Storage service
      oc exec -it $head_pod -- bash -lc "/tmp/bigsql-upgrade-prep.sh <bucket-name>"
      Important: In the output that is produced by the upgrade preparation script, take note of the sentence that mentions the number of worker nodes that you must have in your new Db2 Big SQL installation.
    4. Run the instance stop script.
      oc exec -it $head_pod -- bash -lc "/tmp/bigsql-stop.sh"
  • Scale down the Db2 Big SQL instance to completely stop it.

    Scale down the Db2 Big SQL instance to 0 head and 0 workers to release the worker pod PV.

    oc scale deployment bigsql-ibm-bigsql-dev-engine --replicas=0
    oc scale statefulset worker --replicas=0
    oc scale deployment bigsql-ibm-bigsql-dev-ldap --replicas=0
    oc scale deployment bigsql-ibm-bigsql-dev-metastore-db --replicas=0

Procedure

The upgrade is done by installing the new version of Db2 Big SQL on the upgraded platform, provisioning a new instance of Db2 Big SQL, and mounting the persistent volume that was used by the previous Db2 Big SQL installation to allow its data and metadata to be imported into the new instance.

  1. Follow the procedure to install the latest version of Db2 Big SQL in the upgraded IBM Cloud Pak for Data control plane, and provision a Db2 Big SQL instance.

    The instance must have the same number of worker nodes as the previous version of Db2 Big SQL. After the upgrade is complete, you can add or drop worker nodes. The instance must also be configured to connect to the same remote data source as the Db2 Big SQL deployment that is being upgraded. Finally, make sure to do the relevant post-provisioning tasks for the instance, and that the instance is generally healthy.

  2. Identify the new Db2 Big SQL instance that you created in the previous step, and which will receive the data and metadata from the Db2 Big SQL deployment that is being upgraded, by getting the list of Db2 Big SQL instance IDs.
    ./cpd-cli service-instance list --service-type bigsql --profile <profile-name>
    Note: The profile that is used must be set up with the identity of the user that created the new Db2 Big SQL instance. Otherwise, the instance won't be listed.
  3. Set the INST_ID variable to the instance ID.
    INST_ID=<Db2 Big SQL intance ID>
  4. Attach the PV of the previous version of Db2 Big SQL to the pods of the latest version.
    oc set volume deployment/bigsql-${INST_ID}-head --add -c bigsql-engine -m /mnt/PPV -t persistentVolumeClaim --name previous-pv --claim-name bigsql-engine
    oc set volume statefulset/bigsql-${INST_ID}-worker --add -c bigsql-worker -m /mnt/PPV -t persistentVolumeClaim --name previous-pv --claim-name bigsql-worker-0

    The head and worker pods restart, and as part of the startup the upgrade occurs. You can follow the process by running the following command, which will tail the Db2 Big SQL head pod log as it restarts and the upgrade proceeds.

    head_pod=$(oc get pod -l app=db2-bigsql,instance=$INST_ID,bigsql-node-role=head --no-headers=true -o custom-columns=POD:.metadata.name)
    oc logs -f $head_pod
  5. Verify that the upgrade completed successfully.
  6. If the upgrade failed, retry it.
  7. Check for available patches.
  8. Complete the tasks listed in What to do next.

Verifying that the upgrade completed successfully

When the upgrade is complete and successful, the following message is displayed at regular intervals in the Db2 Big SQL head pod log:

head_pod=$(oc get pod -l app=db2-bigsql,instance=$INST_ID,bigsql-node-role=head --no-headers=true -o custom-columns=POD:.metadata.name)
oc logs -f $head_pod

[...]

Upgrade completed successfully.  Big SQL service is ready for use

To confirm that the upgrade was successful and the cluster is operational, run a smoke test:

head_pod=$(oc get pod -l app=db2-bigsql,instance=$INST_ID,bigsql-node-role=head --no-headers=true -o custom-columns=POD:.metadata.name)

# If connected to a Hadoop cluster 
oc exec -it $head_pod -- bash -lc '/usr/ibmpacks/IBM-Big_SQL/current/bigsql-cli/bigsql-admin -smoke

# If connected exclusively to an Object Store service, you must provide the name of a bucket that exists on the storage service to execute the smoke test
oc exec -it $head_pod -- bash -lc '/usr/ibmpacks/IBM-Big_SQL/current/bigsql-cli/bigsql-admin -smoke -o <bucket-name>

To further validate the operational state of the cluster, you can connect an application or JDBC client to it to execute workloads. However, you must first disable access to the previous Db2 Big SQL instance and enable access to the newly installed one. Note that you are only disabling access to the previous instance. The instance itself and the data it holds remain in the cluster.

To disable access over JDBC to the previous Db2 Big SQL instance, run the following command:

oc delete service bigsql-jdbc

Then enable access to the new instance. You can now connect to the new instance to run workloads.

Retrying the upgrade

If the upgrade fails, and after an analysis of the failure and a corrective step was taken, it is possible to re-initiate the upgrade process by restarting all the pods for the new Db2 Big SQL instance. The following command deletes those pods, which will then be recreated and restarted by the Red Hat OpenShift platform:

oc delete pods -n Project -l app=db2-bigsql,instance=$INST_ID

It is also possible, but more time consuming, to delete the new Db2 Big SQL instance, create a new one as described in step 1 of the Procedure, and resume the upgrade procedure at step 2.

In both cases, the preexisting Db2 Big SQL v6.0.0 deployment is unmodified, and acts as a backup from which the upgrade process can be initiated as many times as needed.

Checking for available patches

Determine whether there are any patches available the version of Db2 Big SQL that you installed:

Clusters connected to the internet
Run the following command to check for patches:
./cpd-cli status \
--repo ./repo.yaml \
--namespace Project \ 
--assembly big-sql \
--patches \
--available-updates 
Air-gapped clusters
See the list of Available patches for Db2 Big SQL.

If you need to apply patches to the service, follow the guidance in Applying a Db2 Big SQL patch.

What to do next

Clean up the following resources.

  • Delete compute and network resources.

    After validating that the upgraded instance is operational, you can delete the previous instance to free up resources on the cluster.

    Note: This action deletes all compute (pod, deployments, StatefulSets, and so on) and network resources (services) associated with the instance. If the recommended step to mark them for retention was not applied, this action also deletes the storage resources that contain the instance data. If marked for retention, the PV resource bigsql-engine is not cleaned up, and you can keep it as a backup after the previous Db2 Big SQL instance is deleted.
  • Clean up PVs.

    If you decide to delete the PVs that holds the previous instance data, you can run the following command to identify the PV to delete:

    # Returns the list of existing PersistentVolume and the name of the PVC that created them
    oc get pv -o custom-columns=NAME:.metadata.name,PVC:.spec.claimRef.name

    You can then delete the PVs that were created by deleting the following Persistent Volume Claims (PVCs) with the oc delete command:

    • bigsql-engine for the head PV
    • bigsql-worker-<number> for the worker PVs

    For example:

    # List the PVs and associated PVCs
    [root@svtpe-inf ~]# oc get pv -o custom-columns=NAME:.metadata.name,PVC:.spec.claimRef.name
    NAME                                       PVC
    pvc-04e91750-9f59-40e7-b077-f878120a5ba4   bigsql-engine
    pvc-05cd31c7-3770-4dca-b274-ea794cd8b239   bigsql-worker-0
    pvc-07bd3f57-aa7e-4736-9f27-cae3d63ce3e7   bigsql-worker-1
    
    # Delete the relevant PVs
    [root@svtpe-inf ~]# oc delete pv pvc-04e91750-9f59-40e7-b077-f878120a5ba4 
    PersistentVolume pvc-04e91750-9f59-40e7-b077-f878120a5ba4 deleted
    [root@svtpe-inf ~]# oc delete pv pvc-05cd31c7-3770-4dca-b274-ea794cd8b239 
    PersistentVolume pvc-05cd31c7-3770-4dca-b274-ea794cd8b239 deleted
    [root@svtpe-inf ~]# oc delete pv pvc-07bd3f57-aa7e-4736-9f27-cae3d63ce3e7 
    PersistentVolume pvc-07bd3f57-aa7e-4736-9f27-cae3d63ce3e7 deleted

Following the upgrade, it is likely that the JDBC or JDBC-TLS port changed. To change the port back to the original port number, do the following steps:

  1. Log in to your OpenShift cluster as a project administrator:
    oc login OpenShift_URL:port
  2. Change to the project where the Cloud Pak for Data control plane is installed:
    oc project Project
  3. Check that the original port number isn't currently in use by another service by running the following command on a node (for example, a master or infrastructure node) in the cluster.
    netstat -nap | grep <original-port-number>
  4. Create a cpd-cli profile.

    To manage Db2 Big SQL instances (by running the cpd-cli service-instance command), you must create a cpd-cli profile. The profile must be set up with the identity of a user that was granted access to the Db2 Big SQL instance. For more information about creating a cpd-cli profile, see Creating a profile to use the cpd-cli management commands.

  5. Identify the Db2 Big SQL instance ID.
    ./cpd-cli service-instance list --service-type bigsql --profile <profile-name>
  6. Back up the current JDBC service definition, replacing <instanceid> with the Db2 Big SQL instance ID that you obtained in the previous step.
    oc get services bigsql-<instanceid>-jdbc -o yaml > bigsql_service_bak.yaml
  7. Make one of the following changes to the JDBC service definition.
    oc edit service bigsql-<instanceid>-jdbc
    • Change
      - name: jdbc
        nodePort: <current-port-number>

      to

      - name: jdbc
        nodePort: <original-port-number>
    • Or change
      - name: jdbc-tls
        nodePort: <current-port-number>

      to

      - name: jdbc-tls
        nodePort: <original-port-number>
  8. After the pod restarts, make sure that it is now listening on the original port.
    netstat -nap | grep <original-port-number>
  9. Try a remote JDBC connection on the original port.