Upgrading watsonx Orchestrate from Version 5.2 to Version 5.3.1

Information you need to complete this task

Review the following information before you upgrade watsonx Orchestrate:

Version requirements

All the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is at Version 5.4.0, you must upgrade watsonx Orchestrate to Version 5.4.0.

Environment variables

The commands in this task use environment variables so that you can run the commands exactly as written.

• If you don't have the script that defines the environment variables, see Setting up installation environment variables.
• To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:

  source ./cpd_vars.sh

Before you begin

This task assumes that the following prerequisites are met:

System requirements

This task assumes that the cluster meets the minimum requirements for watsonx Orchestrate.

Where to find more information
If this task is not complete, see System requirements.

In addition, if you plan to use features that require GPU, ensure that you have the appropriate type and number of GPU for watsonx Orchestrate.

Where to find more information
If this task is not complete, see GPU requirements.

Workstation

This task assumes that the workstation from which you will run the upgrade is set up as a client workstation and has the following command-line interfaces:

• IBM Software Hub CLI: cpd-cli
• OpenShift CLI: oc
• Helm CLI: helm

Where to find more information
If this task is not complete, see Updating client workstations.

Control plane

This task assumes that the IBM Software Hub control plane is upgraded.

Where to find more information
If this task is not complete, see Upgrading an instance of IBM Software Hub.

Private container registry

If your environment uses a private container registry (for example, your cluster is air-gapped), this task assumes that the following tasks are complete:

1. The watsonx Orchestrate software images are mirrored to the private container registry.

   Where to find more information
   If this task is not complete, see Mirroring images to a private container registry.

2. The cpd-cli is configured to pull the olm-utils-v4 image from the private container registry.

   Where to find more information
   If this task is not complete, see Pulling the olm-utils-v4 image from the private container registry.

GPU operators

If you plan to use features that require GPUs, this task assumes that the operators required to use GPUs are installed.

Where to find more information
If this task is not complete, see Installing operators for services that require GPUs.

Red Hat OpenShift AI

If you plan to use features that require Red Hat OpenShift AI, this task assumes that Red Hat OpenShift AI is installed.

Where to find more information
If this task is not complete, see Installing Red Hat OpenShift AI.

Multicloud Object Gateway

This task assumes that Multicloud Object Gateway is upgraded, if needed.

Where to find more information
If this task is not complete, see Upgrading Multicloud Object Gateway.

Red Hat OpenShift Serverless Knative Eventing

This task assumes that the following tasks are complete:

1. Red Hat OpenShift Serverless Knative Eventing is upgraded

   Where to find more information
   If this task is not complete, see Installing Red Hat OpenShift Serverless Knative Eventing.

2. The IBM Events Operator for the instance is upgraded.

   Where to find more information
   If this task is not complete, see Upgrading the IBM Events Operator.

Cluster-scoped resources

This task assumes that the cluster-scoped resources, such as custom resource definitions, cluster roles, and cluster role bindings, were updated.

Where to find more information
If this task is not complete, see Updating the cluster-scoped resources for the platform and services.

Image pull secrets

This task assumes that the secrets that contain the image pull credentials for the instance exist.

Where to find more information
If this task is not complete, see Creating image pull secrets for an instance of IBM Software Hub.

Procedure

Complete the following tasks to upgrade watsonx Orchestrate:

1. Specifying installation options
2. Troubleshooting the issues during upgrade
3. Upgrading the service
4. Validating the upgrade
5. Upgrading the service instance
6. Post upgrade tasks
7. What to do next

Specifying installation options

In IBM Software Hub Version 5.4.0, the default behavior for watsonx Orchestrate is to install only the Agentic features.

Review the following table to determine whether you need to specify installation options :

Custom upgrade options for Agentic assistant mode

If you plan to upgrade the previous versions of watsonx™ Orchestrate with custom upgrade options, specify the appropriate options in a file named install-options.yml in the cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work).

---
# ............................................................................
# watsonx Orchestrate parameters
# ............................................................................
non_olm:
  watsonxOrchestrate:
    installMode: "agentic_assistant"  
    watsonxAI:
      watsonxaiifm: true

Custom upgrade options for Agentic mode

If you plan to upgrade the previous versions of watsonx Orchestrate with custom upgrade options, specify the appropriate options in a file named install-options.yml in the cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work).

---
# ............................................................................
# watsonx Orchestrate parameters
# ............................................................................
non_olm:
  watsonxOrchestrate: 
    watsonxAI:
      watsonxaiifm: true

Troubleshooting the issues during upgrade

For a complete list of troubleshooting information for upgrade issues, see Upgrade troubleshooting.

Upgrading the service

To upgrade watsonx Orchestrate:

1. Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:

   ${CPDM_OC_LOGIN}

   Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.
2. Update the operator and custom resource for watsonx Orchestrate.

   Run the appropriate command to create the custom resource.

   Default installation (without installation options)

   cpd-cli manage install-components \
   --license_acceptance=true \
   --components=watsonx_orchestrate \
   --release=${VERSION} \
   --patch_id=${PATCH_ID} \
   --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
   --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
   --image_pull_prefix=${IMAGE_PULL_PREFIX} \
   --image_pull_secret=${IMAGE_PULL_SECRET} \
   --upgrade=true

   Custom installation (with installation options)

   cpd-cli manage install-components \
   --license_acceptance=true \
   --components=watsonx_orchestrate \
   --release=${VERSION} \
   --patch_id=${PATCH_ID} \
   --operator_ns=${PROJECT_CPD_INST_OPERATORS} \
   --instance_ns=${PROJECT_CPD_INST_OPERANDS} \
   --image_pull_prefix=${IMAGE_PULL_PREFIX} \
   --image_pull_secret=${IMAGE_PULL_SECRET} \
   --param-file=/tmp/work/install-options.yml \
   --upgrade=true

Validating the upgrade

[SUCCESS]... The install-components command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=watsonx_orchestrate

Upgrading the service instance

The service instance is automatically upgraded when you upgrade watsonx Orchestrate.

Post upgrade tasks

Migrating the Agentic Task Manager (ATM) internal endpoints from non‑TLS (HTTP:9044) to TLS‑enabled (HTTPS:9045)

After you upgrade to Version 5.4.x, follow the mentioned steps to migrate the Agentic Task Manager (ATM) internal endpoints from non‑TLS (HTTP:9044) to TLS‑enabled (HTTPS:9045). This ensures proper tool‑flow execution after the upgrade.

1. Login to Red Hat OpenShift cluster.

   oc login <cluster-url>

2. Extract the current ATM server configuration from the Kubernetes secret.

   kubectl get secret wo-agentic-task-manager-server-env \
     -n cpd-instance-1 \
   -o jsonpath='{.data.\.secret\.env}' | base64 --decode | grep SERVER_INTERNAL

   Expected Output:
   Important: Store the value of SERVER_INTERNAL_HOSTNAME for later use. Ensure that the value for SERVER_INTERNAL_PROTOCOL is set to https and SERVER_INTERNAL_PORT is set to 9045.

   SERVER_INTERNAL_PROTOCOL=https
   SERVER_INTERNAL_HOSTNAME=wo-agentic-task-manager.cpd-instance-1.svc.cluster.local
   SERVER_INTERNAL_PORT=9045

3. Create a file named atm_endpoint_tls_migration.sql.
4. Copy the following content in atm_endpoint_tls_migration.sql:

   ---

   atm_endpoint_tls_migration.sql

   -- ============================================================================
   -- Migration: ATM Endpoint TLS Upgrade — ALL TOOLS (5.3.0 -> 5.3.1)
   -- ============================================================================
   
   -- Endpoint config (edit if your namespace differs)
   SET atm_migration.old_url = 'http://<SERVER_INTERNAL_HOSTNAME>:9044';
   SET atm_migration.new_url = 'https://<SERVER_INTERNAL_HOSTNAME>:9045';
   
   BEGIN;
   
   -- ============================================================================
   -- 0. Ensure migration_log table exists
   -- ============================================================================
   CREATE TABLE IF NOT EXISTS migration_log (
       id             SERIAL PRIMARY KEY,
       migration_name TEXT NOT NULL,
       applied_at     TIMESTAMPTZ NOT NULL DEFAULT NOW(),
       description    TEXT,
       details        JSONB,
       CONSTRAINT migration_log_name_unique UNIQUE (migration_name)
   );
   
   -- Idempotency guard
   DO $$
   BEGIN
       IF EXISTS (
           SELECT 1 FROM migration_log
           WHERE migration_name = 'atm_endpoint_tls_migration_5_3_1'
       ) THEN
           RAISE NOTICE '=== Migration atm_endpoint_tls_migration_5_3_1 already applied. Skipping. ===';
           RAISE NOTICE 'To re-run: DELETE FROM migration_log WHERE migration_name = ''atm_endpoint_tls_migration_5_3_1'';';
           RETURN;
       END IF;
   END $$;
   
   -- ============================================================================
   -- 1. Pre-migration snapshot
   -- ============================================================================
   DO $$
   DECLARE
       v_tools_count INT := 0;
       v_tv_count    INT := 0;
       v_total       INT := 0;
   BEGIN
       SELECT COUNT(*) INTO v_tools_count FROM tools
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
   
       SELECT COUNT(*) INTO v_tv_count FROM tool_version
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
   
       v_total := v_tools_count + v_tv_count;
   
       RAISE NOTICE '=== Pre-migration affected rows ===';
       RAISE NOTICE '  tools:        %', v_tools_count;
       RAISE NOTICE '  tool_version: %', v_tv_count;
       RAISE NOTICE '  TOTAL:        %', v_total;
   
       IF v_total = 0 THEN
           RAISE NOTICE 'No rows require migration.';
       END IF;
   END $$;
   
   -- ============================================================================
   -- 2. Migrate: tools.binding (draft — all tools)
   -- ============================================================================
   DO $$
   DECLARE
       v_new_url TEXT := current_setting('atm_migration.new_url');
       v_count   INT := 0;
   BEGIN
       UPDATE tools
       SET binding = REPLACE(
           REPLACE(
               regexp_replace(
                   binding::text,
                   'http://wo-agentic-task-manager[^"]*:9044',
                   v_new_url,
                   'g'
               ),
               '"SERVER_INTERNAL_PROTOCOL":"http"',
               '"SERVER_INTERNAL_PROTOCOL":"https"'
           ),
           '"SERVER_INTERNAL_PORT":"9044"',
           '"SERVER_INTERNAL_PORT":"9045"'
       )::jsonb
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
       GET DIAGNOSTICS v_count = ROW_COUNT;
       RAISE NOTICE 'tools.binding: % rows updated', v_count;
   END $$;
   
   -- ============================================================================
   -- 3. Migrate: tools.binding — targeted jsonb_set for openapi.url
   -- ============================================================================
   DO $$
   DECLARE
       v_count INT := 0;
   BEGIN
       UPDATE tools
       SET binding = jsonb_set(
           binding,
           '{openapi,url}',
           to_jsonb(
               REPLACE(
                   REPLACE(
                       binding->'openapi'->>'url',
                       ':9044', ':9045'
                   ),
                   'http://', 'https://'
               )
           )
       )
       WHERE binding->'openapi'->>'url' LIKE '%wo-agentic-task-manager%'
         AND binding->'openapi'->>'url' LIKE '%9044%';
       GET DIAGNOSTICS v_count = ROW_COUNT;
       IF v_count > 0 THEN
           RAISE NOTICE 'tools.binding (openapi.url targeted): % rows updated', v_count;
       END IF;
   END $$;
   
   -- ============================================================================
   -- 4. Migrate: tool_version.binding (live/versioned — all versions)
   -- ============================================================================
   DO $$
   DECLARE
       v_new_url TEXT := current_setting('atm_migration.new_url');
       v_count   INT := 0;
   BEGIN
       UPDATE tool_version
       SET binding = REPLACE(
           REPLACE(
               regexp_replace(
                   binding::text,
                   'http://wo-agentic-task-manager[^"]*:9044',
                   v_new_url,
                   'g'
               ),
               '"SERVER_INTERNAL_PROTOCOL":"http"',
               '"SERVER_INTERNAL_PROTOCOL":"https"'
           ),
           '"SERVER_INTERNAL_PORT":"9044"',
           '"SERVER_INTERNAL_PORT":"9045"'
       )::jsonb
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
       GET DIAGNOSTICS v_count = ROW_COUNT;
       RAISE NOTICE 'tool_version.binding: % rows updated', v_count;
   END $$;
   
   -- ============================================================================
   -- 5. Migrate: tool_version.binding — targeted jsonb_set for openapi.url
   -- ============================================================================
   DO $$
   DECLARE
       v_count INT := 0;
   BEGIN
       UPDATE tool_version
       SET binding = jsonb_set(
           binding,
           '{openapi,url}',
           to_jsonb(
               REPLACE(
                   REPLACE(
                       binding->'openapi'->>'url',
                       ':9044', ':9045'
                   ),
                   'http://', 'https://'
               )
           )
       )
       WHERE binding->'openapi'->>'url' LIKE '%wo-agentic-task-manager%'
         AND binding->'openapi'->>'url' LIKE '%9044%';
       GET DIAGNOSTICS v_count = ROW_COUNT;
       IF v_count > 0 THEN
           RAISE NOTICE 'tool_version.binding (openapi.url targeted): % rows updated', v_count;
       END IF;
   END $$;
   
   -- ============================================================================
   -- 6. Post-migration verification
   -- ============================================================================
   DO $$
   DECLARE
       v_rem_tools INT := 0;
       v_rem_tv    INT := 0;
       v_total     INT := 0;
   BEGIN
       SELECT COUNT(*) INTO v_rem_tools FROM tools
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
   
       SELECT COUNT(*) INTO v_rem_tv FROM tool_version
       WHERE binding::text LIKE '%wo-agentic-task-manager%'
         AND binding::text LIKE '%:9044%';
   
       v_total := v_rem_tools + v_rem_tv;
   
       RAISE NOTICE '=== Post-migration verification ===';
       RAISE NOTICE '  Remaining in tools:        %', v_rem_tools;
       RAISE NOTICE '  Remaining in tool_version: %', v_rem_tv;
   
       IF v_total > 0 THEN
           RAISE WARNING 'INCOMPLETE: % rows still have old endpoint.', v_total;
       ELSE
           RAISE NOTICE 'COMPLETE: All endpoints updated to TLS (9045) for both draft and live.';
       END IF;
   END $$;
   
   -- ============================================================================
   -- 7. Record migration in audit log
   -- ============================================================================
   DO $$
   DECLARE
       v_old_url TEXT := current_setting('atm_migration.old_url');
       v_new_url TEXT := current_setting('atm_migration.new_url');
   BEGIN
       INSERT INTO migration_log (migration_name, applied_at, description, details)
       VALUES (
           'atm_endpoint_tls_migration_5_3_1',
           NOW(),
           'Upgraded ATM endpoint from non-TLS (http:9044) to TLS (https:9045) for cluster upgrade 5.3.0→5.3.1',
           jsonb_build_object(
               'old_url', v_old_url,
               'new_url', v_new_url,
               'tables', jsonb_build_array('tools', 'tool_version'),
               'column', 'binding'
           )
       )
       ON CONFLICT (migration_name) DO UPDATE
       SET applied_at = NOW(),
           description = EXCLUDED.description,
           details = EXCLUDED.details;
   
       RAISE NOTICE 'Migration recorded as: atm_endpoint_tls_migration_5_3_1';
   END $$;
   
   COMMIT;
   
   

   ---

5. Edit the atm_endpoint_tls_migration.sql file:

   # Edit the configuration section (lines 6-7)
   vi /tmp/atm_endpoint_tls_migration.sql

6. Update the configuration values to match your environment by using the following commands:

   SET atm_migration.old_url = 'http://<SERVER_INTERNAL_HOSTNAME>:9044';
   SET atm_migration.new_url = 'https://<SERVER_INTERNAL_HOSTNAME>:9045';

   Example configuration:

   SET atm_migration.old_url = 'http://wo-agentic-task-manager.cpd-instance-1.svc.cluster.local:9044';
   SET atm_migration.new_url = 'https://wo-agentic-task-manager.cpd-instance-1.svc.cluster.local:9045';

7. To run the migration script on PostgreSQL database:

   POD_NAME=$(oc get pods -l "k8s.enterprisedb.io/instanceName=wo-watson-orchestrate-postgresedb-1" -o jsonpath='{.items[0].metadata.name}')
   DATABASE_NAME="archer"
   
   oc exec -i $POD_NAME -- psql -U postgres -d $DATABASE_NAME < /tmp/atm_endpoint_tls_migration.sql

8. Run the following verification queries to validate the migration status:

   oc exec $POD_NAME -- psql -U postgres -d $DATABASE_NAME -c "
   SELECT COUNT(*) as remaining_tools FROM tools 
   WHERE binding::text LIKE '%wo-agentic-task-manager%' 
   AND binding::text LIKE '%:9044%'; 
   SELECT COUNT(*) as remaining_tool_versions FROM tool_version 
   WHERE binding::text LIKE '%wo-agentic-task-manager%' 
   AND binding::text LIKE '%:9044%';"

   Expected results:

   - `remaining_tools`: 0
   - `remaining_tool_versions`: 0

9. To clean up the migration log, run the following commands:

   oc exec $POD_NAME -- psql -U postgres -d $DATABASE_NAME -c "
   -- Verify migration log entry
   SELECT * FROM migration_log
   WHERE migration_name = 'atm_endpoint_tls_migration_5_3_1';
   "

   Note: Drop the migration_log table only after successful verification.

   oc exec $POD_NAME -- psql -U postgres -d $DATABASE_NAME -c "
   -- Drop the migration_log table 
   DROP TABLE IF EXISTS migration_log;
   "

What to do next