Applying the watsonx Orchestrate 5.2.0 hotfix

Content

Apply this hotfix after completing the watsonx Orchestrate installation for Version 5.2.0, or after you upgrade from previous versions to Version 5.2.0.
This hotfix resolves issues related to the improper functioning of Async OpenAPI tools, import of python file in the WxO agent using ADK and tool-flow output not being displayed.

Set the operator and operand namespaces.

export PROJECT_CPD_INST_OPERATORS=<enter your IBM Software Hub operator project> 
export PROJECT_CPD_INST_OPERANDS=<enter your IBM Software Hub operand project>

Confirm that the watsonx Orchestrate custom resource (CR) in the operands namespace is in Completed status before proceeding.

oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.items[0].status.watsonxOrchestrateStatus}{"\n"}'

Back up the current watsonx Orchestrate CatalogSource, Subscription, and ClusterServiceVersion (CSV).

# CatalogSource
oc get catsrc -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get catsrc {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-catsrc-bkup.yaml

# Subscription
oc get subscriptions.operators.coreos.com -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get subscriptions.operators.coreos.com {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-subscription-bkup.yaml

# CSV
oc get csv -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get csv {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-csv-bkup.yaml

Install skopeo. https://github.com/containers/skopeo/blob/main/install.md
Set the local registry path and use skopeo login to populate a custom auth.json.

export LOCAL_REGISTRY="your_local_registry"
export LOCAL_USER="your_local_registry_username"
export LOCAL_PASS="your_local_registry_password"
export IBM_ENTITLEMENT_KEY="your_ibm_entitlement_key"
export AUTH_JSON_PATH="${HOME}/.airgap/auth.json"

mkdir -p "$(dirname "${AUTH_JSON_PATH}")"

# Populate the custom auth.json using skopeo login
skopeo login cp.icr.io --username cp --password "${IBM_ENTITLEMENT_KEY}" --authfile "${AUTH_JSON_PATH}"
skopeo login "${LOCAL_REGISTRY}" --username "${LOCAL_USER}" --password "${LOCAL_PASS}" --authfile "${AUTH_JSON_PATH}"



Copy watsonx Orchestrate, document processing, and wxo component operator images

# ibm-watsonx-orchestrate-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:adbebd8772c0b7fb8a1c280cbcd6590dba9964cd0075121e149a395fcca071aa \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator@sha256:adbebd8772c0b7fb8a1c280cbcd6590dba9964cd0075121e149a395fcca071aa

# ibm-document-processing-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-document-processing-operator@sha256:390f7724db321e3ef8b9c875b68d071ac3203fd145fa001f0fc876c0142a2949 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-document-processing-operator@sha256:390f7724db321e3ef8b9c875b68d071ac3203fd145fa001f0fc876c0142a2949

# ibm-wxo-component-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-wxo-component-operator@sha256:55b01db9bb99423e9f1e33f1e432b8a1b55c6ceada2b75284c13fdb05246d106 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-wxo-component-operator@sha256:55b01db9bb99423e9f1e33f1e432b8a1b55c6ceada2b75284c13fdb05246d106

# ibm-watsonx-orchestrate-operator CATALOG (FBC)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:876569d7e97585b1af0ac1fb063cbdf5238ba2d804b2dbc3281d2900b616fc11 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:876569d7e97585b1af0ac1fb063cbdf5238ba2d804b2dbc3281d2900b616fc11

# ibm-watsonx-orchestrate-operator BUNDLE
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:02e1d99ca2e712ca7bfeca7979e44e2283cd63ad26626c241c97ca866d70391c \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:02e1d99ca2e712ca7bfeca7979e44e2283cd63ad26626c241c97ca866d70391c




Copy watsonx Orchestrate operands images

# wxo-server-server
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-server@sha256:5838631856a36e4427cd1dad407d813b504caff94bfd2dd1629e4363ef9a3ab5 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-server@sha256:5838631856a36e4427cd1dad407d813b504caff94bfd2dd1629e4363ef9a3ab5

# wxo-server-conversation_controller
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:e15f501e8d4f3a4c7816f69c11af9a474d22426e2e6a92e9c65948382594a270 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:e15f501e8d4f3a4c7816f69c11af9a474d22426e2e6a92e9c65948382594a270

# tools-runtime-manager
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-manager@sha256:8282c5481e08342002d8bba8b78f047866d877d03d743d35745643ed965098aa \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-manager@sha256:8282c5481e08342002d8bba8b78f047866d877d03d743d35745643ed965098aa

# tools-runtime-scheduler
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:9da8b2cf49c2e81411b453aad4110e4e244ab0fd5c5431e9087bc9f4ec695eed \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:9da8b2cf49c2e81411b453aad4110e4e244ab0fd5c5431e9087bc9f4ec695eed

# tools-runtime
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime@sha256:15f937ccd688047cfd1a445df6f46880310ce916841952e1e8a75d4d2d04dece \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime@sha256:15f937ccd688047cfd1a445df6f46880310ce916841952e1e8a75d4d2d04dece

# wo_doc_processing_cache
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wo_doc_processing_cache@sha256:98fd9bb58cdd1a7e412f51eba6715329953d3075a96e1c888294397be3b981bb \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wo_doc_processing_cache@sha256:98fd9bb58cdd1a7e412f51eba6715329953d3075a96e1c888294397be3b981bb

# wo_doc_processing_cache_rds_init
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wo_doc_processing_cache_rds_init@sha256:31862ee97586fcfb4394f5511601825da114651c1d04bd767d89632e96295a9e \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wo_doc_processing_cache_rds_init@sha256:31862ee97586fcfb4394f5511601825da114651c1d04bd767d89632e96295a9e

After copying, verify the images exist in $LOCAL_REGISTRY.

Create a script named 520hotfix.sh with the following contents.

#!/bin/bash
# -----------------------------------------------------------------------------
# Description:
# This script applies the 5.2.0 hotfix updates for Watson Orchestrate components
# running in an OpenShift environment. It performs the following steps:
# 1. Validates that required env vars PROJECT_CPD_INST_OPERATORS and
#    PROJECT_CPD_INST_OPERANDS are set. If not, exits with guidance.
# 2. Takes backups of the Subscription and CSVs before any change.
# 3. Applies hotfix image patches to specific container indices in the CSV.
# 4. Waits for the CSV to reach the "Succeeded" state.
# 5. Deletes the wo-uiproxy ConfigMap, waits for it to be recreated, then
#    restarts the wo-uiproxy deployment and waits for rollout completion.
#    If the ConfigMap does not appear within the timeout, prints guidance
#    to perform the rollout restart manually once it shows up.
# -----------------------------------------------------------------------------

set -euo pipefail

# -----------------------------------------------------------------------------
# User-configurable timeouts
# -----------------------------------------------------------------------------
CSV_WAIT_TIMEOUT="${CSV_WAIT_TIMEOUT:-600}"                 # seconds
CM_RECREATE_TIMEOUT="${CM_RECREATE_TIMEOUT:-12000}"         # seconds
UIPROXY_ROLLOUT_TIMEOUT="${UIPROXY_ROLLOUT_TIMEOUT:-900}"   # seconds

# CSV and job
CSV_NAME="${CSV_NAME:-ibm-watsonx-orchestrate-operator.v6.0.0}"
JOB_NAME="${JOB_NAME:-wo-archer-server-db-schema-job}"

# OLM resources
SUB_NAME="${SUB_NAME:-ibm-cpd-watsonx-orchestrate-operator}"

# UI proxy resources
UIPROXY_CM_NAME="${UIPROXY_CM_NAME:-wo-uiproxy-nginx-mfe-catalog-conf}"
UIPROXY_DEPLOY_NAME="${UIPROXY_DEPLOY_NAME:-wo-uiproxy}"

# -----------------------------------------------------------------------------
# Hotfix images (updated to 5.2.0 digests)
# -----------------------------------------------------------------------------
# ibm-watsonx-orchestrate-operator
WO_OPERATOR_HOTFIX_IMAGE_VALUE="${WO_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:adbebd8772c0b7fb8a1c280cbcd6590dba9964cd0075121e149a395fcca071aa}"

# ibm-document-processing-operator
DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE="${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-document-processing-operator@sha256:390f7724db321e3ef8b9c875b68d071ac3203fd145fa001f0fc876c0142a2949}"

# ibm-wxo-component-operator
COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE="${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-wxo-component-operator@sha256:55b01db9bb99423e9f1e33f1e432b8a1b55c6ceada2b75284c13fdb05246d106}"

# -----------------------------------------------------------------------------
# Helpers
# -----------------------------------------------------------------------------
ts() { date +"%Y-%m-%d %H:%M:%S"; }

require() {
  command -v "$1" >/dev/null 2>&1 || { echo "[$(ts)] Missing required command: $1"; exit 1; }
}

get_cm_uid() {
  local name="$1" ns="$2"
  oc get cm "$name" -n "$ns" -o jsonpath='{.metadata.uid}' 2>/dev/null || true
}

wait_for_csv_succeeded() {
  local csv="$1" ns="$2" timeout_secs="$3"
  local elapsed=0 step=20
  while true; do
    local phase
    phase="$(oc get csv "$csv" -n "$ns" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
    if [[ "$phase" == "Succeeded" ]]; then
      echo "[$(ts)] CSV ${csv} in namespace ${ns} is in Succeeded state."
      break
    fi
    if (( elapsed >= timeout_secs )); then
      echo "[$(ts)] CSV ${csv} in namespace ${ns} did not reach Succeeded within ${timeout_secs}s."
      echo "[$(ts)] Debug: CSV status follows (first 120 lines):"
      oc get csv "$csv" -n "$ns" -o yaml | sed -n '1,120p' || true
      exit 1
    fi
    echo "[$(ts)] Waiting for CSV ${csv} in namespace ${ns}... (${elapsed}/${timeout_secs}s)"
    sleep "$step"
    (( elapsed += step ))
  done
}

wait_for_new_configmap() {
  local name="$1" ns="$2" old_uid="$3" timeout_secs="$4"
  local elapsed=0 step=5
  local ref="configmap/${name} -n ${ns}"

  if [[ -z "$old_uid" ]]; then
    echo "[$(ts)] No existing ${ref} detected. Will wait for first creation."
  else
    echo "[$(ts)] Existing ${ref} UID: ${old_uid}. Will wait for a new UID."
  fi

  while true; do
    local uid
    uid="$(get_cm_uid "$name" "$ns")"
    if [[ -n "$uid" && "$uid" != "$old_uid" ]]; then
      echo "[$(ts)] ${ref} is present with UID: ${uid}"
      return 0
    fi
    if (( elapsed >= timeout_secs )); then
      echo "[$(ts)] Timed out waiting for ${ref} within ${timeout_secs}s."
      echo "[$(ts)] Guidance: Once it appears, run these commands manually:"
      echo "  oc -n ${ns} rollout restart deploy ${UIPROXY_DEPLOY_NAME}"
      echo "  oc -n ${ns} rollout status deploy ${UIPROXY_DEPLOY_NAME} --timeout=${UIPROXY_ROLLOUT_TIMEOUT}s"
      return 1
    fi
    echo "[$(ts)] Waiting for ${ref}... (${elapsed}/${timeout_secs}s)"
    sleep "$step"
    (( elapsed += step ))
  done
}

rollout_restart_and_wait() {
  local deploy="$1" ns="$2" timeout_secs="$3"
  echo "[$(ts)] Triggering rollout restart for deployment/${deploy} -n ${ns}"
  oc -n "$ns" rollout restart deploy "$deploy" 1>/dev/null
  echo "[$(ts)] Waiting for deployment/${deploy} -n ${ns} rollout to complete..."
  oc -n "$ns" rollout status deploy "$deploy" --timeout="${timeout_secs}s"
  echo "[$(ts)] Deployment ${deploy} -n ${ns} rollout completed."
}

# -----------------------------------------------------------------------------
# Validations
# -----------------------------------------------------------------------------
require oc

# Fail if required env vars are not set
if [[ -z "${PROJECT_CPD_INST_OPERATORS:-}" || -z "${PROJECT_CPD_INST_OPERANDS:-}" ]]; then
  echo "[$(ts)] Required environment variables are not set."
  echo "[$(ts)] Please set both variables and re-run:"
  echo "  export PROJECT_CPD_INST_OPERATORS=<operators-namespace>"
  echo "  export PROJECT_CPD_INST_OPERANDS=<operands-namespace>"
  exit 1
fi

# -----------------------------------------------------------------------------
# Section 0: Backups before making changes
# -----------------------------------------------------------------------------
BACKUP_DIR="${BACKUP_DIR:-./wxo-hotfix-backup-$(date +%Y%m%d-%H%M%S)}"
CSV_LABEL_KEY="operators.coreos.com/ibm-watsonx-orchestrate-operator.${PROJECT_CPD_INST_OPERATORS}"

echo "[$(ts)] Creating backup directory at ${BACKUP_DIR}"
mkdir -p "$BACKUP_DIR"

# Backup Subscription
if oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" >/dev/null 2>&1; then
  echo "[$(ts)] Backing up subscriptions.operators.coreos.com/${SUB_NAME} -n ${PROJECT_CPD_INST_OPERATORS}"
  oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o yaml > "${BACKUP_DIR}/subscriptions.operators.coreos.com_${SUB_NAME}.yaml"
else
  echo "[$(ts)] subscriptions.operators.coreos.com/${SUB_NAME} -n ${PROJECT_CPD_INST_OPERATORS} not found. Skipping subscription backup."
fi

# Backup the specific CSV defined by CSV_NAME (if present)
if oc -n "${PROJECT_CPD_INST_OPERATORS}" get csv "${CSV_NAME}" >/dev/null 2>&1; then
  oc -n "${PROJECT_CPD_INST_OPERATORS}" get csv "${CSV_NAME}" -o yaml \
    > "${BACKUP_DIR}/clusterserviceversion.operators.coreos.com_${CSV_NAME}.yaml"
  echo "[$(ts)] clusterserviceversion.operators.coreos.com ${CSV_NAME} backed up to ${BACKUP_DIR}/clusterserviceversion.operators.coreos.com_${CSV_NAME}.yaml"
else
  echo "[$(ts)] clusterserviceversion.operators.coreos.com ${CSV_NAME} not found in namespace ${PROJECT_CPD_INST_OPERATORS}, skipping CSV backup."
fi

# -----------------------------------------------------------------------------
# Section 1: Patch the Watson Orchestrate CSV with updated hotfix component images
# -----------------------------------------------------------------------------
echo "[$(ts)] Deleting job ${JOB_NAME} in namespace ${PROJECT_CPD_INST_OPERANDS} (if present)..."
oc delete job "$JOB_NAME" -n "$PROJECT_CPD_INST_OPERANDS" --ignore-not-found=true

patches=(
  "0 ${WO_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "3 ${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "5 ${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE}"
)

for patch in "${patches[@]}"; do
  IFS=' ' read -r index image <<< "$patch"
  echo "[$(ts)] Patching CSV ${CSV_NAME} deployment index ${index} with image ${image}"
  oc patch csv "$CSV_NAME" \
    --namespace "$PROJECT_CPD_INST_OPERATORS" \
    --type='json' \
    --patch="[
      {\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/$index/spec/template/spec/containers/0/image\", \"value\": \"$image\"}
    ]"
done

echo "[$(ts)] CSV patching completed. Checking CSV status..."
wait_for_csv_succeeded "$CSV_NAME" "$PROJECT_CPD_INST_OPERATORS" "$CSV_WAIT_TIMEOUT"

# -----------------------------------------------------------------------------
# Section 2: Recreate wo-uiproxy ConfigMap and restart wo-uiproxy
# -----------------------------------------------------------------------------
echo "[$(ts)] Preparing to recreate configmap/${UIPROXY_CM_NAME} -n ${PROJECT_CPD_INST_OPERANDS}"
old_uid="$(get_cm_uid "$UIPROXY_CM_NAME" "$PROJECT_CPD_INST_OPERANDS" || true)"
if [[ -n "$old_uid" ]]; then
  echo "[$(ts)] Current configmap/${UIPROXY_CM_NAME} -n ${PROJECT_CPD_INST_OPERANDS} UID: ${old_uid}"
else
  echo "[$(ts)] configmap/${UIPROXY_CM_NAME} -n ${PROJECT_CPD_INST_OPERANDS} not found prior to deletion."
fi

echo "[$(ts)] Deleting configmap/${UIPROXY_CM_NAME} -n ${PROJECT_CPD_INST_OPERANDS} (ignore if not found)"
oc delete cm "$UIPROXY_CM_NAME" -n "$PROJECT_CPD_INST_OPERANDS" --ignore-not-found=true || true

echo "[$(ts)] Waiting for configmap/${UIPROXY_CM_NAME} -n ${PROJECT_CPD_INST_OPERANDS} to be (re)created..."
if wait_for_new_configmap "$UIPROXY_CM_NAME" "$PROJECT_CPD_INST_OPERANDS" "${old_uid:-}" "$CM_RECREATE_TIMEOUT"; then
  echo "[$(ts)] Rolling restart of deployment/${UIPROXY_DEPLOY_NAME} -n ${PROJECT_CPD_INST_OPERANDS} after ConfigMap recreation"
  rollout_restart_and_wait "$UIPROXY_DEPLOY_NAME" "$PROJECT_CPD_INST_OPERANDS" "$UIPROXY_ROLLOUT_TIMEOUT"
else
  echo "[$(ts)] ConfigMap ${UIPROXY_CM_NAME} did not appear in time. Skipping automated rollout restart."
  echo "[$(ts)] Action required: After the ConfigMap exists, run:"
  echo "  oc -n ${PROJECT_CPD_INST_OPERANDS} rollout restart deploy ${UIPROXY_DEPLOY_NAME}"
  echo "  oc -n ${PROJECT_CPD_INST_OPERANDS} rollout status deploy ${UIPROXY_DEPLOY_NAME} --timeout=${UIPROXY_ROLLOUT_TIMEOUT}s"
  exit 1
fi

# -----------------------------------------------------------------------------
# Final message
# -----------------------------------------------------------------------------
echo "------------------------------------------------------------------"
echo "[$(ts)] 5.2.0 Hotfix steps completed."
echo "Backups saved under ${BACKUP_DIR}"
echo "Monitor the Watson Orchestrate CR status by running:"
echo "  oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o yaml"
echo "Ensure that the watsonx Orchestrate CR status is in 'Completed' state."
echo "------------------------------------------------------------------"

Make the script executable and run it.

chmod 775 520hotfix.sh
./520hotfix.sh

It can take several minutes for the operator rollout to complete. Monitor status:

oc get wo -o yaml

The watsonx Orchestrate CR must return to Completed.