Upgrading watsonx Assistant from Version 5.3 to Version 5.4

An instance administrator can upgrade watsonx Assistant from Version 5.3 to Version 5.4.

Who needs to complete this task?

Instance administrator To upgrade watsonx Assistant, you must be an instance administrator. An instance administrator has permission to manage software in the following projects:

The operators project for the instance

The operators for this instance of watsonx Assistant are installed in the operators project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERATORS} environment variable refers to the operators project.

The operands project for the instance

The custom resources for the control plane and watsonx Assistant are installed in the operands project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERANDS} environment variable refers to the operands project.

When do you need to complete this task?

Review the following options to determine whether you need to complete this task:

  • If you want to upgrade the IBM Software Hub control plane and one or more services at the same time, follow the process in Upgrading an instance of IBM Software Hub instead.
  • If you didn't upgrade watsonx Assistant when you upgraded the IBM Software Hub control plane, complete this task to upgrade watsonx Assistant.

    Repeat as needed If you are responsible for multiple instances of IBM Software Hub, you can repeat this task to upgrade more instances of watsonx Assistant on the cluster.

Information you need to complete this task

Review the following information before you upgrade watsonx Assistant:

Version requirements

All the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is at Version 5.4.0, you must upgrade watsonx Assistant to Version 5.4.0.

Environment variables
The commands in this task use environment variables so that you can run the commands exactly as written.
  • If you don't have the script that defines the environment variables, see Setting up installation environment variables.
  • To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:
    source ./cpd_vars.sh
Common core services
watsonx Assistant requires the IBM Software Hub common core services.

If the common core services are not at the correct version in the operands project for the instance, the common core services are automatically upgraded when you upgrade watsonx Assistant. The common core services upgrade increases the amount of time the upgrade takes to complete.

Before you begin

This task assumes that the following prerequisites are met:

System requirements
This task assumes that the cluster meets the minimum requirements for watsonx Assistant.
Where to find more information
If this task is not complete, see System requirements.
In addition, if you plan to use features that require GPU, ensure that you have the appropriate type and number of GPU for watsonx Assistant.
Where to find more information
If this task is not complete, see GPU requirements.
Workstation
This task assumes that the workstation from which you will run the upgrade is set up as a client workstation and has the following command-line interfaces:
  • IBM Software Hub CLI: cpd-cli
  • OpenShift® CLI: oc
  • Helm CLI: helm
Where to find more information
If this task is not complete, see Updating client workstations.
Control plane
This task assumes that the IBM Software Hub control plane is upgraded.
Where to find more information
If this task is not complete, see Upgrading an instance of IBM Software Hub.
Private container registry
If your environment uses a private container registry (for example, your cluster is air-gapped), this task assumes that the following tasks are complete:
  1. The watsonx Assistant software images are mirrored to the private container registry.
    Where to find more information
    If this task is not complete, see Mirroring images to a private container registry.
  2. The cpd-cli is configured to pull the olm-utils-v4 image from the private container registry.
    Where to find more information
    If this task is not complete, see Pulling the olm-utils-v4 image from the private container registry.
GPU operators
If you plan to use features that require GPUs, this task assumes that the operators required to use GPUs are installed.
Where to find more information
If this task is not complete, see Installing operators for services that require GPUs.
Red Hat® OpenShift AI
If you plan to use features that require Red Hat OpenShift AI, this task assumes that Red Hat OpenShift AI is installed.
Where to find more information
If this task is not complete, see Installing Red Hat OpenShift AI.
Multicloud Object Gateway
This task assumes that Multicloud Object Gateway is upgraded, if needed.
Where to find more information
If this task is not complete, see Upgrading Multicloud Object Gateway.
Red Hat OpenShift Serverless Knative Eventing
This task assumes that the following tasks are complete:
  1. Red Hat OpenShift Serverless Knative Eventing is upgraded:
    Where to find more information
    If this task is not complete, see Installing Red Hat OpenShift Serverless Knative Eventing.
  2. The IBM Events Operator for the instance is upgraded:
    Where to find more information
    If this task is not complete, see Upgrading the IBM Events Operator.
Cluster-scoped resources
This task assumes that the cluster-scoped resources, such as custom resource definitions, cluster roles, and cluster role bindings, were updated.
Where to find more information
If this task is not complete, see Updating the cluster-scoped resources for the platform and services.

Procedure

Complete the following tasks to upgrade watsonx Assistant:

  1. Back up your data.

    For more information, see Backing up your data in IBM Software Hub in the watsonx Assistant product documentation.

  2. Backing up and retaining temporary patches
  3. Specifying additional installation options
  4. Upgrading the service
  5. Validating the upgrade
  6. Upgrading existing service instances
  7. What to do next

Backing up and retaining temporary patches

Upgrading watsonx Assistant removes all pre-existing temporary patches that do not have the label 'type=critical-configuration'. If you need to retain some temporary patches, you can preserve them before upgrading by labeling them as critical. For example, you might have a custom configuration that is specific to your environment and was not included in the new release.
Note:
  • Ignore this section if you don't find any pre-existing temporary patches.
  • If you are unsure about what patches need to be retained, contact IBM Support.

To back up and label the patches that need to be retained, follow these steps:

  1. Run the following command to list all temporary patches for your instances. Save the list to the file patches_list.txt.

    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS}   >patches_list.txt
    
  2. Save the YAML file for your existing temporary patches by running the following command:

    oc get temporarypatch -o yaml > old_patch_backups.yaml
    
  3. Get the list of the temporary patches from your deployment to determine which patches you want to preserve.
    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS}
    
  4. Apply or remove the "type": "critical-configuration" label as appropriate:
    • For all patches that you want to retain, use the following command:
      oc label temporarypatch <patch_name> type=critical-configuration
      

      For example:

      oc label temporarypatch wa-store-assistant-limits type=critical-configuration
    • For the patches that you want to remove, use the following command:
      oc label temporarypatch <patch_name> type-
      

      For example:

      oc label temporarypatch wa-store-assistant-limits type-
      
  5. Run the following commands to verify that the temporary patches you want to preserve have the label type=critical-configuration:

    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS} -l type=critical-configuration
    oc get temporarypatch -n ${PROJECT_CPD_INST_OPERANDS} --show-labels

Migrating guidance for models

Note: Complete this task before you upgrade to Version 5.4.x.

To know about the supported and deprecated models in Version 5.4.x, see Supported foundation models for GPU features.

To retain the same model configuration from previous versions
  1. Get the list of models that are to be retained.
    oc get wa wa -o jsonpath='{.configOverrides.ifm.model_config.ootb}' | jq -c 'keys'
    Sample output:
    No output (returns nothing)
    Or
    ["granite-3-8b-instruct","llama-3-1-70b-instruct"]
  2. If no models are returned (empty output)

    Add granite-3-8b-instruct for ootbModels in install-options.yml file.

    ---
    # ............................................................................
    # watsonx Assistant parameters
    # ............................................................................
    non_olm:
      watsonAssistant:
        watsonxAiType: embedded
        ootbModels:
          - granite-3-8b-instruct
  3. If one or more models are returned

    Update the install-options.yml file by replacing <retained-model-name> with the model name or names from the output.

    ---
    # ............................................................................
    # watsonx Assistant parameters
    # ............................................................................
    non_olm:
      watsonAssistant:
        watsonxAiType: embedded
        ootbModels:
          - <retained-model-name>
    
To retain the same models from previous versions and also install a new model
Note: Ensure that you have sufficient GPUs to retain older models while deploying the new model.
Specify the following option for ootbModels in install-options.yml file:
---
# ............................................................................
# watsonx Assistant parameters
# ............................................................................
non_olm:
  watsonAssistant:
    watsonxAiType: embedded
    ootbModels: 
      - gpt-oss-120b
To migrate older models to the new model

If multiple assistants are using older models for Conversational Search, configure extra_vars in the store component to avoid manually updating each assistant.

Example: Migrate a single model

The following OC command uses ibm/granite-3-8b-instruct as an example. Replace the value for CUSTOM_EOL_MODELS with your preferred model.
oc patch wa wa -n ${PROJECT_CPD_INST_OPERANDS} --type=merge --patch='{"configOverrides":{"store":{"extra_vars":{"store": {"CUSTOM_EOL_MODELS": "ibm/granite-3-8b-instruct:openai/gpt-oss-120b"}}}}}'
Example: Migrate multiple models
oc patch wa wa -n ${PROJECT_CPD_INST_OPERANDS} --type=merge --patch='{"configOverrides":{"store":{"extra_vars":{"store": {"CUSTOM_EOL_MODELS": "meta-llama/llama-3-1-70b-instruct:openai/gpt-oss-120b,ibm/granite-3-8b-instruct:openai/gpt-oss-120b"}}}}}'
To remove older models and install new model
Note:
  • Use this approach if GPU capacity is insufficient to retain existing models while deploying the new model.
  • You might experience downtime during this process due to model changes.
  1. Disable IFM.
    oc patch wa wa --type='merge' -p='{"configOverrides":{"enabled_components":{"store":{"ifm":false}},"watsonx_enabled":false}}'
  2. Remove existing models (if any).
    oc patch wa wa --type json --patch '[{ "op": "remove", "path": "/configOverrides/ifm" }]' 2>/dev/null
    Verify the component status.
    oc get wa wa -o jsonpath='{.status.componentStatus.verified}'
    Expected output
    20/20
  3. Disable models (if applicable).
    1. To disable the OOTB model, see Disabling the Out of the Box model.
    2. To disable the specialized model, see Disabling the specialized model.
  4. Upgrade to Version 5.4.x.

    Update the install-options.yml file with gpt-oss-120b in ootbModels value.

    ---
    # ............................................................................
    # watsonx Assistant parameters
    # ............................................................................
    non_olm:
      watsonAssistant:
        watsonxAiType: embedded
        ootbModels: 
          - gpt-oss-120b
  5. Migrate older models after upgrade.

    If multiple assistants are using older models for Conversational Search, configure extra_vars in the store component to avoid manually updating each assistant.

    Example: Migrate a single model

    The following OC command uses ibm/granite-3-8b-instruct as an example. Replace the value for CUSTOM_EOL_MODELS with your preferred model.
    oc patch wa wa -n ${PROJECT_CPD_INST_OPERANDS} --type=merge --patch='{"configOverrides":{"store":{"extra_vars":{"store": {"CUSTOM_EOL_MODELS": "ibm/granite-3-8b-instruct:openai/gpt-oss-120b"}}}}}'
    Example: Migrate multiple models
    oc patch wa wa -n ${PROJECT_CPD_INST_OPERANDS} --type=merge --patch='{"configOverrides":{"store":{"extra_vars":{"store": {"CUSTOM_EOL_MODELS": "meta-llama/llama-3-1-70b-instruct:openai/gpt-oss-120b,ibm/granite-3-8b-instruct:openai/gpt-oss-120b"}}}}}'

Specifying installation options

When you upgrade watsonx Assistant, specify the following option in the install-options.yml file in the work directory.

---
# ............................................................................
# watsonx Assistant parameters
# ............................................................................
non_olm:
  watsonAssistant:
    size: Production
    bigpv: false
    analytics: true
    watsonxAiType: embedded
    ootbModels: 
      - gpt-oss-120b
Property Description
size The deployment size for watsonx Assistant.

The deployment size determines the number of resources allocated to watsonx Assistant.

Default value
Production
Valid values
large
A large deployment has at least 3 replicas of each pod to support production-scale workloads with a large number of concurrent API calls. large is equivalent to the large scaleConfig setting.
Production
A production deployment has at least two replicas of each pod to support production-scale workloads. Production is equivalent to the medium scaleConfig setting.
Starter
A starter deployment has fewer resources and less computing power than a production deployment. Starter is the equivalent to the small scaleConfig setting.

In previous releases, this deployment type was called the development deployment type.

bigpv Specify whether to create larger physical volumes to improve IOPS performance.

Create larger physical volumes if your storage class IOPS performance depends on the size of the physical volume.

Important: You cannot change this setting after you install watsonx Assistant.

You do not need to create larger physical volumes if you use the following storage:

  • Red Hat OpenShift Data Foundation
  • IBM Fusion Data Foundation
  • IBM Fusion Global Data Platform
  • IBM Storage Scale Container Native
  • Portworx
  • IBM Cloud Block Storage
Default value
false
Valid values
false
Create physical volumes with the default size.
true
Create larger physical volumes to improve IOPS performance.
analytics Specify whether to store chat logs and analytics.
Default value
true
Valid values
false
Do not store chat logs and analytics.
true
Store chat logs and analytics.
watsonxAiType Specify this option if you want to install Inference foundation models (watsonx_ai_ifm) to enable the following features, which require GPUs:

Omit this option if you do not want to enable the preceding features.

For more information about supported GPUs, see the GPU requirements for models.

Default value
The default value depends on whether you are installing or upgrading watsonx Assistant:
  • For installations, the default value is none.

    If you omit this option, the GPU features are not enabled.

  • For upgrades, the existing value is used as the default value.

    If you omit this option, the option, the current configuration is used.

Valid values
embedded
Install Inference foundation models (watsonx_ai_ifm) to enable features that require GPUs.
none
Do not install Inference foundation models (watsonx_ai_ifm).

The GPU features will not be enabled.

ootbModels Specify the model to use for features that require GPUs.
Default value
gpt-oss-120b
Valid values
gpt-oss-120b
Note: Ensure that you upgrade watsonx Assistant at the end after all other components are successfully upgraded.

Upgrading the service

To upgrade watsonx Assistant:

  1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:
    ${CPDM_OC_LOGIN}
    Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
  2. Update the operator and custom resource for watsonx Assistant.
    cpd-cli manage install-components \
    --license_acceptance=true \
    --components=watson_assistant \
    --release=${VERSION} \
    --patch_id=${PATCH_ID} \
    --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
    --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
    --image_pull_prefix=${IMAGE_PULL_PREFIX} \
    --image_pull_secret=${IMAGE_PULL_SECRET} \
    --param-file=/tmp/work/install-options.yml \
    --upgrade=true

Validating the upgrade

watsonx Assistant is upgraded when the install-components command returns:
[SUCCESS]... The install-components command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=watson_assistant

Upgrading existing service instances

The service instances are automatically upgraded when you upgrade watsonx Assistant.

What to do next

watsonx Assistant is ready to use. To get started with watsonx Assistant, see Administering watsonx Assistant.