Upgrading watsonx Assistant from Version 4.8 to Version 5.1

An instance administrator can upgrade watsonx Assistant from IBM Cloud Pak® for Data Version 4.8 to IBM Software Hub Version 5.1.

Who needs to complete this task?

Instance administrator To upgrade watsonx Assistant, you must be an instance administrator. An instance administrator has permission to manage software in the following projects:

The operators project for the instance

The operators for this instance of watsonx Assistant are installed in the operators project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERATORS} environment variable refers to the operators project.

The operands project for the instance

The custom resources for the control plane and watsonx Assistant are installed in the operands project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERANDS} environment variable refers to the operands project.

When do you need to complete this task?

Review the following options to determine whether you need to complete this task:

  • If you want to upgrade the IBM Software Hub control plane and one or more services at the same time, follow the process in Upgrading an instance of IBM Software Hub instead.
  • If you didn't upgrade watsonx Assistant when you upgraded the IBM Software Hub control plane, complete this task to upgrade watsonx Assistant.

    Repeat as needed If you are responsible for multiple instances of IBM Software Hub, you can repeat this task to upgrade more instances of watsonx Assistant on the cluster.

Information you need to complete this task

Review the following information before you upgrade watsonx Assistant:

Version requirements

All the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is at Version 5.1.3, you must upgrade watsonx Assistant to Version 5.1.3.

Environment variables
The commands in this task use environment variables so that you can run the commands exactly as written.
  • If you do not have the script that defines the environment variables, see Setting up installation environment variables.
  • To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:
    source ./cpd_vars.sh
Common core services
watsonx Assistant requires the IBM Software Hub common core services.

If the common core services are not at the correct version in the operands project for the instance, the common core services are automatically upgraded when you upgrade watsonx Assistant. The common core services upgrade increases the amount of time the upgrade takes to complete.

Before you begin

This task assumes that the following prerequisites are met:

Prerequisite Where to find more information
The cluster meets the minimum requirements for watsonx Assistant. If this task is not complete, see System requirements.
The workstation from which you will run the upgrade is set up as a client workstation and has the following command-line interfaces:
  • IBM Software Hub CLI: cpd-cli
  • OpenShift® CLI: oc
If this task is not complete, see Updating client workstations.
The IBM Software Hub control plane is upgraded. If this task is not complete, see Upgrading an instance of IBM Software Hub.
For environments that use a private container registry, such as air-gapped environments, the watsonx Assistant software images are mirrored to the private container registry. If this task is not complete, see Mirroring images to a private container registry.
For environments that use a private container registry, such as air-gapped environments, the cpd-cli is configured to pull the olm-utils-v3 image from the private container registry. If this task is not complete, see Pulling the olm-utils-v3 image from the private container registry.
If you plan to use features that require GPUs, the operators required to use GPUs are installed. If this task is not complete, see Installing operators for services that require GPUs.
Red Hat® OpenShift AI is installed. If this task is not complete, see Installing Red Hat OpenShift AI.
Multicloud Object Gateway is upgraded, if needed. If this task is not complete, see Upgrading Multicloud Object Gateway.
Red Hat OpenShift Serverless Knative Eventing is upgraded, if needed. If this task is not complete, see Installing Red Hat OpenShift Serverless Knative Eventing.

Procedure

Complete the following tasks to upgrade watsonx Assistant:

  1. Back up your data.

    For more information, see Backing up your data in IBM Software Hub in the watsonx Assistant product documentation.

  2. Backing up and retaining temporary patches
  3. Specifying additional installation options
  4. Upgrading the service
  5. Validating the upgrade
  6. Validating the EDB Postgres upgrade
  7. Cleaning up resources
  8. Upgrading existing service instances
  9. What to do next

Specifying installation options

When you upgrade watsonx Assistant, specify the following option in the install-options.yml file in the work directory.

################################################################################
# watsonx Assistant parameters
################################################################################
#watson_assistant_size: Production
#watson_assistant_bigpv: false
#watson_assistant_analytics_enabled: true
#watson_assistant_watsonx_ai_type: embedded
#watson_assistant_syom_models: []
#watson_assistant_ootb_models: []
Property Description
watson_assistant_size The deployment size for watsonx Assistant.

The deployment size determines the number of resources allocated to watsonx Assistant.

Default value
Production
Valid values
large
A large deployment has at least 3 replicas of each pod to support production-scale workloads with a large number of concurrent API calls. large is equivalent to the large scaleConfig setting.
Production
A production deployment has at least two replicas of each pod to support production-scale workloads. Production is equivalent to the medium scaleConfig setting.
Starter
A starter deployment has fewer resources and less computing power than a production deployment. Starter is the equivalent to the small scaleConfig setting.

In previous releases, this deployment type was called the development deployment type.

watson_assistant_bigpv Specify whether to create larger physical volumes to improve IOPS performance.

Create larger physical volumes if your storage class IOPS performance depends on the size of the physical volume.

Important: You cannot change this setting after you install watsonx Assistant.

You do not need to create larger physical volumes if you use the following storage:

  • Red Hat OpenShift Data Foundation
  • IBM Fusion Data Foundation
  • IBM Fusion Global Data Platform
  • IBM Storage Scale Container Native
  • Portworx
  • IBM Cloud Block Storage
Default value
false
Valid values
false
Create physical volumes with the default size.
true
Create larger physical volumes to improve IOPS performance.
watson_assistant_analytics_enabled Specify whether to store chat logs and analytics.
Default value
true
Valid values
false
Do not store chat logs and analytics.
true
Store chat logs and analytics.
watson_assistant_watsonx_ai_type Specify this option if you want to install Inference foundation models (watsonx_ai_ifm) to enable the following features, which require GPUs:

Omit this option if you do not want to enable the preceding features.

For more information about supported GPUs, see the Hardware requirements for watsonx Assistant.

Default value
The default value depends on whether you are installing or upgrading watsonx Assistant:
  • For installations, the default value is none.

    If you omit this option, the GPU features are not enabled.

  • For upgrades, the existing value is used as the default value.

    If you omit this option, the option, the current configuration is used.

Valid values
embedded
Install Inference foundation models (watsonx_ai_ifm) to enable features that require GPUs.
none
Do not install Inference foundation models (watsonx_ai_ifm).

The GPU features will not be enabled.

watson_assistant_syom_models 5.1.2 and later This option is available starting in IBM Software Hub Version 5.1.2.
Specify whether you want to use a specialized model that is specifically tuned for use with watsonx Assistant for:
Important: The following models will be automatically installed if you install Inference foundation models (watson_assistant_watsonx_ai_type: embedded) and you do not specify a value for watson_assistant_syom_models or watson_assistant_ootb_models:
  • ibm-granite-8b-unified-api-model-v2
  • granite-3-8b-instruct
Default value
[]
Valid values
[]
Do not install a specialized model.
ibm-granite-8b-unified-api-model-v2
Install the ibm-granite-8b-unified-api-model-v2 specialized model. This model enables assistants to:
  • Rewrite user questions to an understood format for conversational search
  • Gather information to fill in variables in a conversational skill
Including this parameter
Ensure that you uncomment the following line and specify the model name as a list item on a new line:
watson_assistant_syom_models:
  - ibm-granite-8b-unified-api-model-v2
watson_assistant_ootb_models 5.1.2 and later This option is available starting in IBM Software Hub Version 5.1.2.
Specify whether you want to use a general model for:
Important: The following models will be automatically installed if you install Inference foundation models (watson_assistant_watsonx_ai_type: embedded) and you do not specify a value for watson_assistant_syom_models or watson_assistant_ootb_models:
  • ibm-granite-8b-unified-api-model-v2
  • granite-3-8b-instruct
Default value
[]
Valid values
[]
Do not install a general model.
granite-3-8b-instruct
Install the granite-3-8b-instruct general model. This model enables assistants to:
  • Answer conversational search questions
Important: If you use a private container registry, you must explicitly mirror the granite-3-8b-instruct image to the private container registry.
llama-3-1-70b-instruct
Install the llama-3-1-70b-instruct general model. This model enables assistants to:
  • Rewrite user questions to an understood format for conversational search
  • Answer conversational search questions.
  • Gather information to fill in variables in a conversational skill
Important: If you use a private container registry, you must explicitly mirror the llama-3-1-70b-instruct image to the private container registry.
Including this parameter
Ensure that you uncomment the following line and specify the model name as a list item on a new line.
Install the granite-3-8b-instruct model
watson_assistant_ootb_models:
  - granite-3-8b-instruct
Install the llama-3-1-70b-instruct model
watson_assistant_ootb_models:
  - llama-3-1-70b-instruct
Install both models
watson_assistant_ootb_models:
  - granite-3-8b-instruct
  - llama-3-1-70b-instruct

Backing up and retaining temporary patches

Upgrading watsonx Assistant removes all pre-existing temporary patches that do not have the label 'type=critical-configuration'. If you need to retain some temporary patches, you can preserve them before upgrading by labeling them as critical. For example, you might have a custom configuration that is specific to your environment and was not included in the new release.

Note:
  • Ignore this section if you don't find any pre-existing temporary patches.
  • If you are unsure about what patches need to be retained, contact IBM Support.

To back up and label the patches that need to be retained, follow these steps:

  1. Run the following command to list all temporary patches for your instances. Save the list to the file patches_list.txt.

    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS}   >patches_list.txt
    
  2. Save the YAML file for your existing temporary patches by running the following command:

    oc get temporarypatch -o yaml > old_patch_backups.yaml
    
  3. Get the list of the temporary patches from your deployment to determine which patches you want to preserve.
    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS}
    
  4. Apply or remove the "type": "critical-configuration" label as appropriate:
    • For all patches that you want to retain, use the following command:
      oc label temporarypatch <patch_name> type=critical-configuration
      

      For example:

      oc label temporarypatch wa-store-assistant-limits type=critical-configuration
    • For the patches that you want to remove, use the following command:
      oc label temporarypatch <patch_name> type-
      

      For example:

      oc label temporarypatch wa-store-assistant-limits type-
      
  5. Run the following commands to verify that the temporary patches you want to preserve have the label type=critical-configuration:

    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS} -l type=critical-configuration
    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS} --show-labels

Upgrading the service

Important: The Operator Lifecycle Manager (OLM) objects for watsonx Assistant were updated when you upgraded the IBM Software Hub platform. The cpd-cli manage apply-olm updates all of the OLM objects in the operators project at the same time.

To upgrade watsonx Assistant:

  1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:
    ${CPDM_OC_LOGIN}
    Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
  2. Update the custom resource for watsonx Assistant.
    cpd-cli manage apply-cr \
    --components=watson_assistant \
    --release=${VERSION} \
    --cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
    --param-file=/tmp/work/install-options.yml \
    --license_acceptance=true \
    --upgrade=true

Validating the upgrade

watsonx Assistant is upgraded when the apply-cr command returns:
[SUCCESS]... The apply-cr command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=watson_assistant

Validating the EDB Postgres upgrade

When do you need to complete this task?
Complete this task after you successfully upgraded watsonx Assistant to Version 5.1.
To verify the successful EDB Postgres migration from Version 12 to Version 16:
  1. Set an instance environment variable.
    export INSTANCE=wa
  2. Make sure wa instance goes to a fully verified state.
    oc get wa 
    NAME   VERSION    READY   READYREASON   UPDATING   UPDATINGREASON   DEPLOYED   VERIFIED   QUIESCE        AGE
    wa     5.1.0      True    Stable        False      Stable           19/19      19/19      NOT_QUIESCED   25h
  3. Check if $INSTANCE-postgres-16 cluster is in healthy or ready state.
    oc get cluster $INSTANCE-postgres-16
    NAME             AGE   INSTANCES   READY   STATUS                     PRIMARY
    wa-postgres-16   22h   3           3       Cluster in healthy state   wa-postgres-16-1
    oc get pods -l app=$INSTANCE-postgres-16
    NAME               READY   STATUS    RESTARTS   AGE
    wa-postgres-16-1   1/1     Running   0          22h
    wa-postgres-16-2   1/1     Running   0          22h
    wa-postgres-16-3   1/1     Running   0          22h
  4. Check whether all the jobs are in completed state.
    oc get jobs -l slot=$INSTANCE
    NAME                                                   COMPLETIONS   DURATION   AGE
    wa-5-1-0-update-schema-store-db-job   1/1           2m22s      23h
    wa-cleanup-job                                         1/1           23h        23h
    wa-create-bucket-store-cos-job                         1/1           12s        25h
    wa-create-slot-job                                     1/1           2m38s      25h
    wa-create-slot-store-db-job                            1/1           42s        25h
  5. Check whether the store cronjob is pointing to the new EDB Postgres-16 cluster.
    oc get cronjob $INSTANCE-store-cronjob  -o yaml | grep postgres
                  # If today is the weekly backup day specified in postgres.backup.history.files.weeklyBackupDay then write the backup to the weekly dir
                  value: wa-postgres-16-rw.cpd.svc
                      name: wa-postgres-16-admin-auth
                      name: wa-postgres-16-admin-auth
  6. If wa-postgres-16-rw.cpd.svc points to old EDB Postgres cluster instead of EDB Postgres-16, then delete the cronjob with the following command. This helps the operator to create a new cronjob with latest config.
    oc delete cronjob $INSTANCE-store-cronjob
  7. Login to IBM Cloud Pak for Data console.
  8. Test the upgradation by sending a message to one of the old instances:
    • Click Launch tool and click one of the watsonx Assistant service instances.
    • Type a message in the preview chat window to trigger the actions that are defined for the watsonx Assistant before upgrade.
    • Check whether you received the expected response without training the watsonx Assistant again.

Cleaning up resources

Who needs to complete this task?
You must be an administrator of the project where watsonx Assistant is installed.
When do you need to complete this task?
Complete this task to clean up resources after you upgrade watsonx Assistant to Version 5.1.

To delete the store-cronjob:

  1. Use the following command to delete the store-cronjob:
    oc delete cronjob ${INSTANCE}-store-cronjob
  2. Verify that the job is deleted successfully:
    oc get cronjob -n ${PROJECT_CPD_INST_OPERANDS}
    After a few minutes, a new job with the same name must reappear.
To clean up the old EDB Postgres Version 12 cluster after upgrade:
  1. Set an instance environment variable.
    export INSTANCE=wa
  2. Delete old EDB Postgres Version 12 cluster.
    oc delete cluster -l app=$INSTANCE-postgres,component=postgres,service=conversation,slot=$INSTANCE
  3. Delete old certificate created for EDB Postgres Version 12 cluster.
    oc delete cert -l component=cert-postgres,service=conversation,slot=$INSTANCE
  4. Delete old secrets created by the preceding certificate.
    oc delete secret -l component=cert-postgres,service=conversation,slot=$INSTANCE
  5. Delete the watsonx Assistant created old secrets.
    oc delete secret -l app=$INSTANCE-postgres,component=postgres,service=conversation

Upgrading existing service instances

The service instances are automatically upgraded when you upgrade watsonx Assistant.

What to do next

watsonx Assistant is ready to use. To get started with watsonx Assistant, see Administering watsonx Assistant.