Upgrading watsonx Orchestrate from Version 5.2 to Version 5.3.0

An instance administrator can upgrade watsonx Orchestrate from Version 5.2 to Version 5.3.0.

Note: The new agentic experience will not be available in existing tenants when you upgrade to Version 5.3.0 from a previous version that uses the skill-based experience. However, any new tenants that you create in the Version 5.3.0 cluster will have the new agentic experience only.
Warning: If you are upgrading to version 5.3.0 from a release that includes Agentic skills assistant features, you must set the installMode parameter to retain those features. If you don’t set this parameter, the upgrade fails. For more information, see Specifying installation options.
Who needs to complete this task?

Instance administrator To upgrade watsonx Orchestrate, you must be an instance administrator. An instance administrator has permission to manage software in the following projects:

The operators project for the instance

The operators for this instance of watsonx Orchestrate are installed in the operators project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERATORS} environment variable refers to the operators project.

The operands project for the instance

The custom resources for the control plane and watsonx Orchestrate are installed in the operands project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERANDS} environment variable refers to the operands project.

When do you need to complete this task?

Review the following options to determine whether you need to complete this task:

  • If you want to upgrade the IBM Software Hub control plane and one or more services at the same time, follow the process in Upgrading an instance of IBM Software Hub instead.
  • If you didn't upgrade watsonx Orchestrate when you upgraded the IBM Software Hub control plane, complete this task to upgrade watsonx Orchestrate.

    Repeat as needed If you are responsible for multiple instances of IBM Software Hub, you can repeat this task to upgrade more instances of watsonx Orchestrate on the cluster.

Information you need to complete this task

Review the following information before you upgrade watsonx Orchestrate:

Version requirements

All the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is at Version 5.4.0, you must upgrade watsonx Orchestrate to Version 5.4.0.

Environment variables
The commands in this task use environment variables so that you can run the commands exactly as written.
  • If you don't have the script that defines the environment variables, see Setting up installation environment variables.
  • To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:
    source ./cpd_vars.sh

Before you begin

This task assumes that the following prerequisites are met:

System requirements
This task assumes that the cluster meets the minimum requirements for watsonx Orchestrate.
Where to find more information
If this task is not complete, see System requirements.
In addition, if you plan to use features that require GPU, ensure that you have the appropriate type and number of GPU for watsonx Orchestrate.
Where to find more information
If this task is not complete, see GPU requirements.
Workstation
This task assumes that the workstation from which you will run the upgrade is set up as a client workstation and has the following command-line interfaces:
  • IBM Software Hub CLI: cpd-cli
  • OpenShift® CLI: oc
  • Helm CLI: helm
Where to find more information
If this task is not complete, see Updating client workstations.
Control plane
This task assumes that the IBM Software Hub control plane is upgraded.
Where to find more information
If this task is not complete, see Upgrading an instance of IBM Software Hub.
Private container registry
If your environment uses a private container registry (for example, your cluster is air-gapped), this task assumes that the following tasks are complete:
  1. The watsonx Orchestrate software images are mirrored to the private container registry.
    Where to find more information
    If this task is not complete, see Mirroring images to a private container registry.
  2. The cpd-cli is configured to pull the olm-utils-v4 image from the private container registry.
    Where to find more information
    If this task is not complete, see Pulling the olm-utils-v4 image from the private container registry.
GPU operators
If you plan to use features that require GPUs, this task assumes that the operators required to use GPUs are installed.
Where to find more information
If this task is not complete, see Installing operators for services that require GPUs.
Red Hat® OpenShift AI
If you plan to use features that require Red Hat OpenShift AI, this task assumes that Red Hat OpenShift AI is installed.
Where to find more information
If this task is not complete, see Installing Red Hat OpenShift AI.
Multicloud Object Gateway
This task assumes that Multicloud Object Gateway is upgraded, if needed.
Where to find more information
If this task is not complete, see Upgrading Multicloud Object Gateway.
Red Hat OpenShift Serverless Knative Eventing
This task assumes that the following tasks are complete:
  1. Red Hat OpenShift Serverless Knative Eventing is upgraded
    Where to find more information
    If this task is not complete, see Installing Red Hat OpenShift Serverless Knative Eventing.
  2. The IBM Events Operator for the instance is upgraded.
    Where to find more information
    If this task is not complete, see Upgrading the IBM Events Operator.
Cluster-scoped resources
This task assumes that the cluster-scoped resources, such as custom resource definitions, cluster roles, and cluster role bindings, were updated.
Where to find more information
If this task is not complete, see Updating the cluster-scoped resources for the platform and services.
Image pull secrets
This task assumes that the secrets that contain the image pull credentials for the instance exist.
Where to find more information
If this task is not complete, see Creating image pull secrets for an instance of IBM Software Hub.

Procedure

Complete the following tasks to upgrade watsonx Orchestrate:

  1. Specifying installation options
  2. Handling Events Operator dependency
  3. Troubleshooting the issues during upgrade
  4. Upgrading the service
  5. Validating the upgrade
  6. Post upgrade tasks
  7. Upgrading the service instance
  8. What to do next

Specifying installation options

In IBM Software Hub Version 5.4.0, the default behavior for watsonx Orchestrate is to install only the Agentic features.

Review the following table to determine whether you need to specify installation options:

Current feature configuration Target feature configuration Required upgrade actions and parameters
Agentic Agentic
  1. You do not need to specify any installation parameter.
  2. If you want to retain previously used local IFM models, refer to Upgrading from Agentic mode of previous versions.
Agentic skills assistant Agentic
  1. You do not need to specify any installation parameter.
  2. If you want to retain previously used local IFM models, refer to Upgrading from Agentic mode of previous versions.
Agentic skills assistant Agentic assistant When you upgrade watsonx Orchestrate, you must specify the installMode:agentic_skills_assistant parameter. For more details, refer to Upgrading from Agentic skills assistant of previous versions.
Agentic skills assistant Agentic skills assistant When you upgrade watsonx Orchestrate, you must specify the installMode:agentic_skills_assistant parameter. For more details, refer to Upgrading from Agentic skills assistant of previous versions.

Upgrading from Agentic skills assistant mode of previous versions

If you plan to upgrade watsonx™ Orchestrate from Agentic skills assistant of previous versions, specify the appropriate options in a file named install-options.yml in the cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work) and add the following entry:
Note: If you have configured local models using Inference Foundation Models in previous versions and want to retain the same configuration in Version 5.3.0, you must set watsonxaiifm parameter to true.
---
# ............................................................................
# watsonx Orchestrate parameters
# ............................................................................
non_olm:
  watsonxOrchestrate:
    installMode: "agentic_skills_assistant"  
    watsonxAI:
      watsonxaiifm: true

Upgrading from Agentic mode of previous versions

If you plan to upgrade watsonx Orchestrate from Agentic mode of previous versions, specify the appropriate upgrade options in a file named install-options.yml in the cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work) and add the following entry:
Note: If you have configured local models using Inference Foundation Models in previous versions and want to retain the same configuration in Version 5.3.0, you must set watsonxaiifm parameter to true.
---
# ............................................................................
# watsonx Orchestrate parameters
# ............................................................................
non_olm:
  watsonxOrchestrate: 
    watsonxAI:
      watsonxaiifm: true

Handling Events Operator dependency

The Events Operator is exclusively packaged and distributed through OLM. You require an additional workaround to install or replace this component.
Note: Complete the following steps before you proceed to upgrade your service.
  1. Do the pre-upgrade cleanup:
    oc delete rolebinding ibm-lakehouse-leader-election-rolebinding -n ${PROJECT_CPD_INST_OPERATORS} || true
    oc delete role ibm-uab-ads-operator-role -n ${PROJECT_CPD_INST_OPERATORS} || true
  2. Run the install-components command with the preview flag to generate the helm migration command for watsonx Orchestrate.
    cpd-cli manage install-components \
    --license_acceptance=true \
    --components=watsonx_orchestrate \
    --release=${VERSION} \
    --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
    --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
    --image_pull_prefix=${IMAGE_PULL_PREFIX} \
    --image_pull_secret=${IMAGE_PULL_SECRET} \
    --param-file=/tmp/work/install-options.yml \
    --upgrade=true \
    --preview=true
  3. Set the correct path for your cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work). Get the helm upgrade command for the watsonx Orchestrate migration.
    cat $cpdcli_workspace/preview.sh | grep -E "watsonx-orchestrate-migration|watson-assistant-migration"
  4. Run the helm upgrade command that is obtained from Step 3 for watsonx Orchestrate and watsonx Assistant. Refer to the following examples:
    Note: Run the commands only from within your OLM container.
    podman exec -it olm-utils-play-v4 helm upgrade --install --namespace ${PROJECT_CPD_INST_OPERANDS} watsonx-orchestrate \
    /tmp/work/offline/5.3.0/.ibm-pak/data/cases/ibm-watsonx-orchestrate/7.0.0/charts/watsonx-orchestrate-migration-0.0.0.tgz \
    --take-ownership --debug \
    -f /tmp/work/olm-utils-ansible-log/override_file_*.yaml
    podman exec -it olm-utils-play-v4 helm upgrade --install --namespace ${PROJECT_CPD_INST_OPERANDS} watson-assistant \
    /tmp/work/offline/5.3.0/.ibm-pak/data/cases/ibm-watson-assistant/5.10.0/charts/watson-assistant-migration-0.0.0.tgz \
    --take-ownership --debug \
    -f /tmp/work/olm-utils-ansible-log/override_file_*.yaml \
    --set-json global.crAdoption=false
  5. To verify that the helm label is added for watsonx Orchestrate:
    oc get wo wo -n ${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}'
    Expected output:
    Helm
  6. To verify that watsonx Assistant is managed by watsonx Orchestrate operator:
    oc get wa wo-wa -n ${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.metadata.labels.app\.kubernetes\.io/managed-by}'
    Expected output:
    ibm-watson-orchestrate-operator

Troubleshooting the issues during upgrade

Issue Solution
Kafka failure during upgrade See Kafka does not enter the verifying state during upgrade.
Bootstrap job failure during failure See Bootstrap job fails to complete during upgrade or installation.
When local IFM models are configured in previous version Apply the latest hot fix for version 5.3.x during the upgrade process to ensure a smooth completion.
Note: Ensure that you upgrade watsonx Orchestrate at the end after all other components are successfully upgraded.

Upgrading the service

To upgrade watsonx Orchestrate:

  1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:
    ${CPDM_OC_LOGIN}
    Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
  2. Update the operator and custom resource for watsonx Orchestrate.

    Run the appropriate command to create the custom resource.

    Default installation (without installation options)
    cpd-cli manage install-components \
    --license_acceptance=true \
    --components=watsonx_orchestrate \
    --release=${VERSION} \
    --patch_id=${PATCH_ID} \
    --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
    --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
    --image_pull_prefix=${IMAGE_PULL_PREFIX} \
    --image_pull_secret=${IMAGE_PULL_SECRET} \
    --upgrade=true
    Custom installation (with installation options)
    cpd-cli manage install-components \
    --license_acceptance=true \
    --components=watsonx_orchestrate \
    --release=${VERSION} \
    --patch_id=${PATCH_ID} \
    --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
    --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
    --image_pull_prefix=${IMAGE_PULL_PREFIX} \
    --image_pull_secret=${IMAGE_PULL_SECRET} \
    --param-file=/tmp/work/install-options.yml \
    --upgrade=true

Validating the upgrade

watsonx Orchestrate is upgraded when the install-components command returns:
[SUCCESS]... The install-components command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=watsonx_orchestrate

Post upgrade tasks

Apply Hot Fix
You must apply the latest hot fix for version 5.3.x.
To switch to Agentic mode from Agentic skills assistant mode
Edit the watsonx Orchestrate custom resource to Switch to agentic mode.
To switch to Agentic assistant mode from Agentic skills assistant mode
Edit the watsonx Orchestrate custom resource to Switch to agentic assistant mode.

Upgrading the service instance

The service instance is automatically upgraded when you upgrade watsonx Orchestrate.

What to do next

  1. watsonx Orchestrate is ready to use. For information about how to give people access to your instance, see Adding users to watsonx Orchestrate on IBM Software Hub.
  2. You can configure IFM models, enable specific models or disable unused models. For more information, see Configuring IFM models after install or upgrade.
  3. You can add, display, modify, or delete the external models by using external AI Gateway. For more information, see Registering external models through AI gateway.
  4. To enable Intelligent Document Processing (IDP), see [Optional]Enabling Intelligent Document Processing.