IBM Support

Applying the watsonx Orchestrate 5.2.1 hotfix (Hotfix 1)

Fix Readme


Abstract

Apply this hotfix after completing the watsonx Orchestrate installation for Version 5.2.1, or after upgrading from a previous release to Version 5.2.1.

Content

1) Set the operator and operand namespaces

Export operator and operand namespace Set the operator and operand namespaces.
  
export PROJECT_CPD_INST_OPERATORS=<enter your IBM Software Hub operator project>
export PROJECT_CPD_INST_OPERANDS=<enter your IBM Software Hub operand project>

2) Verify watsonx Orchestrate CR status

Confirm that the watsonx Orchestrate custom resource (CR) in the operands namespace shows Completed status before proceeding.
oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.items[0].status.watsonxOrchestrateStatus}{"\n"}'

3) Back up the CatalogSource, Subscription, and CSV

Create backups of the current watsonx Orchestrate CatalogSource, Subscription, and ClusterServiceVersion (CSV).
# CatalogSource
oc get catsrc -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get catsrc {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-catsrc-bkup.yaml

# Subscription
oc get subscriptions.operators.coreos.com -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get subscriptions.operators.coreos.com {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-subscription-bkup.yaml

# CSV
oc get csv -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
  | xargs -r -I {} oc get csv {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
  > watson-orchestrate-csv-bkup.yaml

4) Mirror images to a private Docker registry (air‑gap only)

Install skopeo. https://github.com/containers/skopeo/blob/main/install.md

Set your local registry path and use skopeo login to populate a custom auth.json.

export LOCAL_REGISTRY="your_local_registry"
export LOCAL_USER="your_local_registry_username"
export LOCAL_PASS="your_local_registry_password"
export IBM_ENTITLEMENT_KEY="your_ibm_entitlement_key"
export AUTH_JSON_PATH="${HOME}/.airgap/auth.json"

mkdir -p "$(dirname "${AUTH_JSON_PATH}")"

# Populate the custom auth.json using skopeo login
skopeo login cp.icr.io --username cp --password "${IBM_ENTITLEMENT_KEY}" --authfile "${AUTH_JSON_PATH}"
skopeo login "${LOCAL_REGISTRY}" --username "${LOCAL_USER}" --password "${LOCAL_PASS}" --authfile "${AUTH_JSON_PATH}"


Copy watsonx Orchestrate, document processing, digital employee and wxo component operator images

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:5216d518a0d85bca4cd9a4d21c2456ccab7f547511997c2857850a8294b5424e \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator@sha256:5216d518a0d85bca4cd9a4d21c2456ccab7f547511997c2857850a8294b5424e

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
 docker://icr.io/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642 \
 docker://$LOCAL_REGISTRY/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642


skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
 docker://icr.io/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4 \
 docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
 docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:2436970771d7d8d24c2396718743cd5470a7d43401f5052eb744c5f6f49c0e56 \
 docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:2436970771d7d8d24c2396718743cd5470a7d43401f5052eb744c5f6f49c0e56

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
 docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:5f07b7092f7af3caaf097a2c05be64ae83c575e4dbbbd417f28af156977e5a2e \
 docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:5f07b7092f7af3caaf097a2c05be64ae83c575e4dbbbd417f28af156977e5a2e


Copy watsonx Orchestrate operands images

# wxo-server-server

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-server@sha256:fc1d0a7cce2808e422631e11fbeef60af76227b29f630a86f107d31708bf2997 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-server@sha256:fc1d0a7cce2808e422631e11fbeef60af76227b29f630a86f107d31708bf2997


# wxo-server-conversation_controller
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:8c3757fef8bf8b76f95e17133fe4d0898ab8efe226611f403be90bfddc6efb39 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:8c3757fef8bf8b76f95e17133fe4d0898ab8efe226611f403be90bfddc6efb39

# tools-runtime-manager
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-manager@sha256:de7d96470eb2868c79e85d3fe0419571a8926ce97960793b8435bda0ce7286aa \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-manager@sha256:de7d96470eb2868c79e85d3fe0419571a8926ce97960793b8435bda0ce7286aa

# tools-runtime-scheduler
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:62afd6d3a5114ed975a8e40f2eea2f3642b69ef6a81776ff06ed90176a086a63 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:62afd6d3a5114ed975a8e40f2eea2f3642b69ef6a81776ff06ed90176a086a63

# tools-runtime
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime@sha256:745bb03a4e71f9c93421bf3a9819cfd363979c790f79893d110e0b2aa3bb6e6b \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime@sha256:745bb03a4e71f9c93421bf3a9819cfd363979c790f79893d110e0b2aa3bb6e6b

# ibm-watsonx-orchestrate-onprem-utils

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/ibm-watsonx-orchestrate-onprem-utils@sha256:76baafdd811f2bab5c632d38272f3f4cb05f416d3bf819db4d5288ece3c7afaf \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ibm-watsonx-orchestrate-onprem-utils@sha256:76baafdd811f2bab5c632d38272f3f4cb05f416d3bf819db4d5288ece3c7afaf

# wxo-flow-runtime

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-flow-runtime@sha256:0fe1ff660ba27db1e52fd5329a5c6371b83b6dadf2c5c4857416bf9f33af6389 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-flow-runtime@sha256:0fe1ff660ba27db1e52fd5329a5c6371b83b6dadf2c5c4857416bf9f33af6389

# wxo-builder

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-builder@sha256:1bfd141ce05ba261f1dbffe6e5133813774c7ee71bca98c0170b037b33e45ae3 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-builder@sha256:1bfd141ce05ba261f1dbffe6e5133813774c7ee71bca98c0170b037b33e45ae3

# ads-restapi 

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389

# ads-parsing

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c

# Platformui

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/platformui@sha256:8e17cac7f2076dcbc5b7a32d0237077f972c719eda3ccc4e5afabf0959d26054 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/platformui@sha256:8e17cac7f2076dcbc5b7a32d0237077f972c719eda3ccc4e5afabf0959d26054

# Landing login

skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/landing-login@sha256:4fa19705f7feb82e0ba19b8d44c9b04701c8c6ddcce2b6071fc32cc6f0bcf21c \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/landing-login@sha256:4fa19705f7feb82e0ba19b8d44c9b04701c8c6ddcce2b6071fc32cc6f0bcf21c

After copying, verify the images exist in $LOCAL_REGISTRY.

5) Apply the 5.2.1 Hotfix

Create a script named hotfix521.sh with the following contents.
#!/bin/bash
# -----------------------------------------------------------------------------
# Description:
# This script applies the 5.2.1 hotfix updates for Watson Orchestrate components
# running in an OpenShift environment. It performs the following steps:
# 1. Validates that required env vars PROJECT_CPD_INST_OPERATORS and
#    PROJECT_CPD_INST_OPERANDS are set. If not, exits with guidance.
# 2. Takes backups of the Subscription and CSVs before any change.
# 3. Applies hotfix image patches to specific container indices in the CSV.
# 4. Waits for the CSV to reach the "Succeeded" state.

# -----------------------------------------------------------------------------

set -euo pipefail

# -----------------------------------------------------------------------------
# User-configurable timeouts
# -----------------------------------------------------------------------------
CSV_WAIT_TIMEOUT="${CSV_WAIT_TIMEOUT:-600}"                 # seconds
CM_RECREATE_TIMEOUT="${CM_RECREATE_TIMEOUT:-12000}"         # seconds
JOB_TIMEOUT="${JOB_TIMEOUT:-12000}"		            # seconds
# CSV and job
CSV_NAME="${CSV_NAME:-ibm-watsonx-orchestrate-operator.v6.1.0}"
JOB_NAME="${JOB_NAME:-wo-archer-server-db-schema-job}"

# OLM resources
SUB_NAME="${SUB_NAME:-ibm-cpd-watsonx-orchestrate-operator}"


# Hotfix images
WO_OPERATOR_HOTFIX_IMAGE_VALUE="${WO_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:5216d518a0d85bca4cd9a4d21c2456ccab7f547511997c2857850a8294b5424e}"
DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE="${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c}"
COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE="${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506}"
DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE="${DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4}"
BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE="${BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642}"
ADS_PARSING_IMAGE_VALUE="${ADS_PARSING_IMAGE_VALUE:-cp.icr.io/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c}"
ADS_RESTAPI_IMAGE_VALUE="${ADS_RESTAPI_IMAGE_VALUE:-cp.icr.io/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389}"
# -----------------------------------------------------------------------------
# Helpers
# -----------------------------------------------------------------------------
ts() { date +"%Y-%m-%d %H:%M:%S"; }

require() {
  command -v "$1" >/dev/null 2>&1 || { echo "[$(ts)] Missing required command: $1"; exit 1; }
}


wait_for_csv_succeeded() {
  local csv="$1" ns="$2" timeout_secs="$3"
  local elapsed=0 step=20
  while true; do
    local phase
    phase="$(oc get csv "$csv" -n "$ns" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
    if [[ "$phase" == "Succeeded" ]]; then
      echo "[$(ts)] CSV ${csv} in namespace ${ns} is in Succeeded state."
      break
    fi
    if (( elapsed >= timeout_secs )); then
      echo "[$(ts)] CSV ${csv} in namespace ${ns} did not reach Succeeded within ${timeout_secs}s."
      echo "[$(ts)] Debug: CSV status follows (first 120 lines):"
      oc get csv "$csv" -n "$ns" -o yaml | sed -n '1,120p' || true
      exit 1
    fi
    echo "[$(ts)] Waiting for CSV ${csv} in namespace ${ns}... (${elapsed}/${timeout_secs}s)"
    sleep "$step"
    (( elapsed += step ))
  done
}


rollout_restart_and_wait() {
  local deploy="$1" ns="$2" timeout_secs="$3"
  echo "[$(ts)] Triggering rollout restart for deployment/${deploy} -n ${ns}"
  oc -n "$ns" rollout restart deploy "$deploy" 1>/dev/null
  echo "[$(ts)] Waiting for deployment/${deploy} -n ${ns} rollout to complete..."
  oc -n "$ns" rollout status deploy "$deploy" --timeout="${timeout_secs}s"
  echo "[$(ts)] Deployment ${deploy} -n ${ns} rollout completed."
}

# -----------------------------------------------------------------------------
# Validations
# -----------------------------------------------------------------------------
require oc

# Fail if required env vars are not set
if [[ -z "${PROJECT_CPD_INST_OPERATORS:-}" || -z "${PROJECT_CPD_INST_OPERANDS:-}" ]]; then
  echo "[$(ts)] Required environment variables are not set."
  echo "[$(ts)] Please set both variables and re-run:"
  echo "  export PROJECT_CPD_INST_OPERATORS=<operators-namespace>"
  echo "  export PROJECT_CPD_INST_OPERANDS=<operands-namespace>"
  exit 1
fi

# -----------------------------------------------------------------------------
# Section 0: Backups before making changes
# -----------------------------------------------------------------------------
BACKUP_DIR="${BACKUP_DIR:-./wxo-hotfix-backup-$(date +%Y%m%d-%H%M%S)}"
CSV_LABEL_KEY="operators.coreos.com/ibm-watsonx-orchestrate-operator.${PROJECT_CPD_INST_OPERATORS}"

echo "[$(ts)] Creating backup directory at ${BACKUP_DIR}"
mkdir -p "$BACKUP_DIR"

# Backup Subscription
if oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" >/dev/null 2>&1; then
  echo "[$(ts)] Backing up subscriptions.operators.coreos.com/${SUB_NAME} -n ${PROJECT_CPD_INST_OPERATORS}"
  oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o yaml > "${BACKUP_DIR}/subscriptions.operators.coreos.com_${SUB_NAME}.yaml"
else
  echo "[$(ts)] subscriptions.operators.coreos.com/${SUB_NAME} -n ${PROJECT_CPD_INST_OPERATORS} not found. Skipping subscription backup."
fi

# Backup the specific CSV defined by CSV_NAME (if present)
if oc -n "${PROJECT_CPD_INST_OPERATORS}" get csv "${CSV_NAME}" >/dev/null 2>&1; then
  oc -n "${PROJECT_CPD_INST_OPERATORS}" get csv "${CSV_NAME}" -o yaml \
    > "${BACKUP_DIR}/clusterserviceversion.operators.coreos.com_${CSV_NAME}.yaml"
  echo "[$(ts)] clusterserviceversion.operators.coreos.com ${CSV_NAME} backed up to ${BACKUP_DIR}/clusterserviceversion.operators.coreos.com_${CSV_NAME}.yaml"
else
  echo "[$(ts)] clusterserviceversion.operators.coreos.com ${CSV_NAME} not found in namespace ${PROJECT_CPD_INST_OPERATORS}, skipping CSV backup."
fi


# -----------------------------------------------------------------------------
# Section 1: Patch the Watson Orchestrate CSV with updated hotfix component images
# -----------------------------------------------------------------------------

patches=(
  "0 ${WO_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "3 ${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "5 ${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "1 ${BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE}"
  "2 ${DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE}"
)

for patch in "${patches[@]}"; do
  IFS=' ' read -r index image <<< "$patch"
  echo "[$(ts)] Patching CSV ${CSV_NAME} deployment index ${index} with image ${image}"
  oc patch csv "$CSV_NAME" \
    --namespace "$PROJECT_CPD_INST_OPERATORS" \
    --type='json' \
    --patch="[
      {\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/$index/spec/template/spec/containers/0/image\", \"value\": \"$image\"}
    ]"
done


oc patch csv "$CSV_NAME" \
    --namespace "$PROJECT_CPD_INST_OPERATORS" \
    --type='json' \
    --patch="[
		{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/4/spec/template/spec/containers/0/env/8/value\", \"value\": \"$ADS_RESTAPI_IMAGE_VALUE\"},
    		{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/4/spec/template/spec/containers/0/env/9/value\", \"value\": \"$ADS_PARSING_IMAGE_VALUE\"}
         
    ]"
echo "Patching completed. Checking CSV status..."



echo "[$(ts)] CSV patching completed. Checking CSV status..."
wait_for_csv_succeeded "$CSV_NAME" "$PROJECT_CPD_INST_OPERATORS" "$CSV_WAIT_TIMEOUT"


get_job_uid() {
  local name="$1" ns="$2"
  oc get job "$name" -n "$ns" -o jsonpath='{.metadata.uid}' 2>/dev/null || true
}

wait_for_new_job() {
  local name="$1" ns="$2" old_uid="$3" timeout_secs="$4"
  local elapsed=0 step=5
  local ref="job/${name} -n ${ns}"

  if [[ -z "$old_uid" ]]; then
    echo "[$(ts)] No existing ${ref} detected. Will wait for first creation."
  else
    echo "[$(ts)] Existing ${ref} UID: ${old_uid}. Will wait for a new UID."
  fi

  while true; do
    local uid
    uid="$(get_job_uid "$name" "$ns")"
    if [[ -n "$uid" && "$uid" != "$old_uid" ]]; then
      echo "[$(ts)] ${ref} is present with UID: ${uid}"
      return 0
    fi
    if (( elapsed >= timeout_secs )); then
      echo "[$(ts)] Timed out waiting for ${ref} within ${timeout_secs}s."
      echo "[$(ts)] Guidance: Once it appears, run these commands manually:"
      echo "  oc -n ${ns} delete ${JOB_NAME}"
      return 1
    fi
    echo "[$(ts)] Waiting for ${ref}... (${elapsed}/${timeout_secs}s)"
    sleep "$step"
    (( elapsed += step ))
  done
}


wait_for_job_success() {
  local job_name=$1
  local namespace=$2
  local timeout=${3:-12000}  # default 5 minutes
  local interval=5         # check every 5 seconds

  echo "Waiting for Job '$job_name' in namespace '$namespace' to succeed (timeout: $timeout sec)..."

  SECONDS=0
  while true; do
    status=$(oc get job "$job_name" -n "$namespace" -o jsonpath='{.status.succeeded}' 2>/dev/null)

    if [[ "$status" == "1" ]]; then
      echo "Job '$job_name' succeeded."
      return 0
    fi

    failed=$(oc get job "$job_name" -n "$namespace" -o jsonpath='{.status.failed}' 2>/dev/null)
    if [[ "$failed" != "" && "$failed" -gt 0 ]]; then
      echo " Job '$job_name' failed."
      return 1
    fi

    if (( SECONDS >= timeout )); then
      echo "Timeout reached waiting for Job '$job_name'."
      return 2
    fi

    sleep $interval
  done
}



echo "[$(ts)] Preparing to recreate job/${JOB_NAME} -n ${PROJECT_CPD_INST_OPERANDS}"
old_uid="$(get_job_uid "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" || true)"
if [[ -n "$old_uid" ]]; then
  echo "[$(ts)] Current job/${JOB_NAME} -n ${PROJECT_CPD_INST_OPERANDS} UID: ${old_uid}"
else
  echo "[$(ts)] job/${JOB_NAME} -n ${PROJECT_CPD_INST_OPERANDS} not found prior to deletion."
fi

echo "[$(ts)] Deleting job/${JOB_NAME} -n ${PROJECT_CPD_INST_OPERANDS} (ignore if not found)"

job=$(oc get jobs -n "$PROJECT_CPD_INST_OPERANDS" -o name | grep '^job.batch/wo-archer-server-db-schema' || true)
echo "Job is  $job"
if [ -n "$job" ]; then
  oc delete -n "$PROJECT_CPD_INST_OPERANDS" $job --ignore-not-found
fi


echo " Deleting pods"

pods=$(oc get pods -n "$PROJECT_CPD_INST_OPERANDS" -o name | grep '^pod/wo-archer-server-db-schema' || true)

if [ -n "$pods" ]; then
  echo "Deleting pods: $pods"
  oc delete -n "$PROJECT_CPD_INST_OPERANDS" $pods --ignore-not-found
else
  echo "No matching pods found to delete."
fi


echo " Entering waiting stage"

echo "[$(ts)] Waiting for job/${JOB_NAME} -n ${PROJECT_CPD_INST_OPERANDS} to be (re)created..."
if wait_for_new_job "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" "${old_uid:-}" "$CM_RECREATE_TIMEOUT"; then
  echo "Waiting to be job to be completed"
  wait_for_job_success "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" "$JOB_TIMEOUT"
else
  echo "[$(ts)] Job ${JOB_NAME} did not appear in time. Skipping automated actions."
  echo "[$(ts)] Action required: After the job exists, run:"
  echo "  oc -n ${PROJECT_CPD_INST_OPERANDS} delete job ${JOB_NAME}"
  echo "  oc -n ${PROJECT_CPD_INST_OPERANDS} get ${JOB_NAME} -o jsonpath='{.status.succeeded}'"
  exit 1
fi


# -----------------------------------------------------------------------------
# Final message
# -----------------------------------------------------------------------------
echo "------------------------------------------------------------------"
echo "[$(ts)] 5.2.1 Hotfix steps completed."
echo "Backups saved under ${BACKUP_DIR}"
echo "Monitor the Watson Orchestrate CR status by running:"
echo "  oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o yaml"
echo "Ensure that the watsonx Orchestrate CR status is in 'Completed' state."
echo "------------------------------------------------------------------"
Make the script executable
chmod 775 hotfix521.sh
As the script can take up to an hour to complete, run it with nohup in the background.

Run the script:

nohup sh hotfix521.sh &

Watch progress:

tail -f nohup.out

Operator rollouts can take several minutes. To monitor status:

oc get wo -o yaml
The watsonx Orchestrate CR must return to Completed.

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSVAUS","label":"IBM watsonx Orchestrate Cartridge for IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8mKe0000008OVAIA2","label":"Operate-\u003EInstall"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":""}]

Document Information

Modified date:
10 October 2025

UID

ibm17243990

Page Feedback