Fix Readme
Abstract
Apply this hotfix after completing the watsonx Orchestrate installation for Version 5.2.1, or after upgrading from a previous release to Version 5.2.1.
Content
1) Set the operator and operand namespaces
Export operator and operand namespaces.
export PROJECT_CPD_INST_OPERATORS=<enter your IBM Software Hub operator project>
export PROJECT_CPD_INST_OPERANDS=<enter your IBM Software Hub operand project>
2) Verify watsonx Orchestrate CR status
Confirm that the watsonx Orchestrate custom resource (CR) in the operands namespace shows
oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o jsonpath='{.items[0].status.watsonxOrchestrateStatus}{"\n"}'
3) Back up the CatalogSource, Subscription, and CSV
Create backups of the current watsonx Orchestrate CatalogSource, Subscription, and ClusterServiceVersion (CSV).
# CatalogSource
oc get catsrc -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' | xargs -r -I {} oc get catsrc {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml > watson-orchestrate-catsrc-bkup.yaml
# Subscription
oc get subscriptions.operators.coreos.com -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' | xargs -r -I {} oc get subscriptions.operators.coreos.com {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml > watson-orchestrate-subscription-bkup.yaml
# CSV
oc get csv -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' | xargs -r -I {} oc get csv {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml > watson-orchestrate-csv-bkup.yaml
4) Mirror images to a private Docker registry (air gap only)
Install skopeo: https://github.com/containers/skopeo/blob/main/install.md
Set your local registry path and use skopeo login to populate a custom auth.json.
export LOCAL_REGISTRY="your_local_registry"
export LOCAL_USER="your_local_registry_username"
export LOCAL_PASS="your_local_registry_password"
export IBM_ENTITLEMENT_KEY="your_ibm_entitlement_key"
export AUTH_JSON_PATH="${HOME}/.airgap/auth.json"
mkdir -p "$(dirname "${AUTH_JSON_PATH}")"
# Populate the custom auth.json using skopeo login
skopeo login cp.icr.io --username cp --password "${IBM_ENTITLEMENT_KEY}" --authfile "${AUTH_JSON_PATH}"
skopeo login "${LOCAL_REGISTRY}" --username "${LOCAL_USER}" --password "${LOCAL_PASS}" --authfile "${AUTH_JSON_PATH}"
Copy operator images
# ibm-watsonx-orchestrate-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:5cec4b02221b4875ba02e2bd94fbb46b3f9e08683a91a67d0820ed14e17199d9 docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator@sha256:5cec4b02221b4875ba02e2bd94fbb46b3f9e08683a91a67d0820ed14e17199d9
# ibm-watsonx-orchestrate-operator-catalog
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:dd2229942ca63811855317ab40ced6e773ce10f5bb7e42895317322911101ce3 docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:dd2229942ca63811855317ab40ced6e773ce10f5bb7e42895317322911101ce3
# ibm-watsonx-orchestrate-operator-bundle
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:9f75c85073075f64a977d3ccfc3f6c6bddc134eb6bd6360cdc49574aff09823c docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:9f75c85073075f64a977d3ccfc3f6c6bddc134eb6bd6360cdc49574aff09823c
# ibm-watsonx-orchestrate-ads-operator-controller
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-watsonx-orchestrate-ads-operator-controller@sha256:6f3bc8e980f4159b84bbb1733afba1bfbb4504c8b4353c908c24cfb17296030b docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-ads-operator-controller@sha256:6f3bc8e980f4159b84bbb1733afba1bfbb4504c8b4353c908c24cfb17296030b
# ibm-document-processing-operator (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c docker://${LOCAL_REGISTRY}/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c
# ibm-wxo-component-operator (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506 docker://${LOCAL_REGISTRY}/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506
# ba-saas-uab-wf-operator (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642 docker://$LOCAL_REGISTRY/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642
# ibm-watsonx-orchestrate-digital-employee-operator (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://icr.io/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4 docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4
Copy operand images
# wxo-server-server
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-server@sha256:355ea62432c221fa735c382cc2aa1a16e31cc3de35d750308fbf0db6d081312b docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-server@sha256:355ea62432c221fa735c382cc2aa1a16e31cc3de35d750308fbf0db6d081312b
# wxo-server-conversation_controller
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:86f85a2b2b7bc2f65785950fb86210c9d2ad36a0ed281254858410763a0b01c2 docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:86f85a2b2b7bc2f65785950fb86210c9d2ad36a0ed281254858410763a0b01c2
# ads-tools
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/ads-tools@sha256:5a3f94ad4e1929c52bef2c97530aa36bb8f373148a70dee3c5ed3e63f0fbc3e1 docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ads-tools@sha256:5a3f94ad4e1929c52bef2c97530aa36bb8f373148a70dee3c5ed3e63f0fbc3e1
# ads-restapi (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389 docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389
# ads-parsing (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c
# platformui (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/platformui@sha256:8e17cac7f2076dcbc5b7a32d0237077f972c719eda3ccc4e5afabf0959d26054 docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/platformui@sha256:8e17cac7f2076dcbc5b7a32d0237077f972c719eda3ccc4e5afabf0959d26054
# landing-login (unchanged)
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false docker://cp.icr.io/cp/watsonx-orchestrate/landing-login@sha256:4fa19705f7feb82e0ba19b8d44c9b04701c8c6ddcce2b6071fc32cc6f0bcf21c docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/landing-login@sha256:4fa19705f7feb82e0ba19b8d44c9b04701c8c6ddcce2b6071fc32cc6f0bcf21c
After copying, verify the images exist in $LOCAL_REGISTRY.
Create a script named hotfix521.sh with the following contents.
#!/bin/bash
# -----------------------------------------------------------------------------
# Watson Orchestrate 5.2.1 Hotfix 2
# - Backs up OLM resources
# - Updates CatalogSource image and restarts catalog pod
# - Waits for CatalogSource readiness via connectionState and service endpoints
# - Patches CSV with hotfix operator images and ADS env values
# - Labels the WO CR with hotfix=2
# - Recreates and waits for DB schema job to complete
# -----------------------------------------------------------------------------
set -euo pipefail
# -----------------------------
# Configurable timeouts
# -----------------------------
CSV_WAIT_TIMEOUT="${CSV_WAIT_TIMEOUT:-600}"
CM_RECREATE_TIMEOUT="${CM_RECREATE_TIMEOUT:-12000}"
JOB_TIMEOUT="${JOB_TIMEOUT:-12000}"
CATSRC_READY_TIMEOUT="${CATSRC_READY_TIMEOUT:-900}" # wait for connectionState READY
SVC_ENDPOINTS_TIMEOUT="${SVC_ENDPOINTS_TIMEOUT:-600}" # wait for service endpoints
POD_APPEAR_TIMEOUT="${POD_APPEAR_TIMEOUT:-900}" # optional pod appearance
POD_READY_TIMEOUT="${POD_READY_TIMEOUT:-600}" # optional pod ready
# -----------------------------
# Required namespaces
# -----------------------------
: "${PROJECT_CPD_INST_OPERATORS:?Set PROJECT_CPD_INST_OPERATORS}"
: "${PROJECT_CPD_INST_OPERANDS:?Set PROJECT_CPD_INST_OPERANDS}"
# -----------------------------
# OLM resources and images
# -----------------------------
CSV_NAME="${CSV_NAME:-ibm-watsonx-orchestrate-operator.v6.1.0}"
SUB_NAME="${SUB_NAME:-ibm-cpd-watsonx-orchestrate-operator}"
CATSRC_NAME="${CATSRC_NAME:-ibm-watsonx-orchestrate-catalog}"
# Use mirror path for air gap if needed
CATSRC_IMAGE_VALUE="${CATSRC_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:dd2229942ca63811855317ab40ced6e773ce10f5bb7e42895317322911101ce3}"
# Hotfix operator images
WO_OPERATOR_HOTFIX_IMAGE_VALUE="${WO_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:5cec4b02221b4875ba02e2bd94fbb46b3f9e08683a91a67d0820ed14e17199d9}"
DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE="${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-document-processing-operator@sha256:c71b04d765b386e8baf739e139d5654369d8271f97e61af4b40b0337c8365d0c}"
COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE="${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-wxo-component-operator@sha256:52bb7ffb7453ef7df5f376d7c245f6b6130a1e66bcf76fe0b015075e615b7506}"
DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE="${DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-digital-employee-operator@sha256:36d7e9d9294ab245954b2ea378df3e07c92b3c3401866e912241682f226deac4}"
BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE="${BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE:-icr.io/cpopen/ba-saas-uab-wf-operator@sha256:d4c2a3f274a26dfc26bac9282ed94bac3ef1138f0a6a1ed6d8ba1f2a393a1642}"
# ADS env image values
ADS_PARSING_IMAGE_VALUE="${ADS_PARSING_IMAGE_VALUE:-cp.icr.io/cp/watsonx-orchestrate/ads-parsing@sha256:4b9769b9e1946e3df05a8dd65bb27a325e59ef050da0b79ad049be69edf5098c}"
ADS_RESTAPI_IMAGE_VALUE="${ADS_RESTAPI_IMAGE_VALUE:-cp.icr.io/cp/watsonx-orchestrate/ads-restapi@sha256:c93089a467232d7945c9093c2fe8a19c5e51705d32b5ce5c4d98d998a2543389}"
# Job to re-run
JOB_NAME="${JOB_NAME:-wo-archer-server-db-schema-job}"
# -----------------------------
# Helpers
# -----------------------------
ts() { date +"%Y-%m-%d %H:%M:%S"; }
require() { command -v "$1" >/dev/null 2>&1 || { echo "[$(ts)] Missing required command: $1"; exit 1; }; }
wait_for_csv_succeeded() {
local csv="$1" ns="$2" timeout_secs="$3"
local elapsed=0 step=20
while true; do
local phase
phase="$(oc get csv "$csv" -n "$ns" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
if [[ "$phase" == "Succeeded" ]]; then
echo "[$(ts)] CSV ${csv} in namespace ${ns} is in Succeeded state."
break
fi
if (( elapsed >= timeout_secs )); then
echo "[$(ts)] CSV ${csv} in namespace ${ns} did not reach Succeeded within ${timeout_secs}s."
oc get csv "$csv" -n "$ns" -o yaml | sed -n '1,120p' || true
exit 1
fi
echo "[$(ts)] Waiting for CSV ${csv} in namespace ${ns}... (${elapsed}/${timeout_secs}s)"
sleep "$step"; (( elapsed += step ))
done
}
# Wait for CatalogSource to be READY and its service endpoints to have addresses.
wait_for_catalog_ready() {
local ns="$1" catsrc="$2" cs_timeout="${3:-900}" svc_timeout="${4:-600}"
local elapsed=0 step=5
echo "[$(ts)] Waiting for CatalogSource/${catsrc} in ns ${ns} to reach connectionState READY (timeout ${cs_timeout}s)..."
while true; do
local state msg
state="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.connectionState.lastObservedState}' 2>/dev/null || true)"
msg="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.connectionState.message}' 2>/dev/null || true)"
[[ -n "$state" ]] && echo "[$(ts)] CatalogSource state: ${state} ${msg:-}"
if [[ "$state" == "READY" ]]; then
break
fi
if (( elapsed >= cs_timeout )); then
echo "[$(ts)] Timeout: CatalogSource did not reach READY within ${cs_timeout}s."
oc -n "$ns" get catsrc "$catsrc" -o yaml | sed -n '1,180p' || true
return 1
fi
sleep "$step"; (( elapsed += step ))
done
# Discover OLM-published registry service
local svc_ns svc_name
svc_ns="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.registryService.serviceNamespace}' 2>/dev/null || true)"
svc_name="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.registryService.serviceName}' 2>/dev/null || true)"
if [[ -z "$svc_ns" || -z "$svc_name" ]]; then
echo "[$(ts)] WARNING: registryService not populated yet, will still try to wait on pods if they appear."
else
echo "[$(ts)] Waiting for Endpoints ${svc_name} in ns ${svc_ns} to have at least one address (timeout ${svc_timeout}s)..."
elapsed=0
while true; do
# count addresses on Endpoints object
local addr_count
addr_count="$(oc -n "$svc_ns" get endpoints "$svc_name" -o jsonpath='{range .subsets[*]}{range .addresses[*]}X{end}{end}' 2>/dev/null | wc -c | tr -d ' ')"
if [[ -n "$addr_count" && "$addr_count" -ge 1 ]]; then
echo "[$(ts)] Service endpoints are populated."
break
fi
if (( elapsed >= svc_timeout )); then
echo "[$(ts)] Timeout: service endpoints not ready within ${svc_timeout}s."
oc -n "$svc_ns" get svc "$svc_name" -o yaml | sed -n '1,160p' || true
oc -n "$svc_ns" get endpoints "$svc_name" -o yaml | sed -n '1,160p' || true
# do not hard-fail here. Proceed to optional pod wait.
break
fi
sleep "$step"; (( elapsed += step ))
done
fi
# Optional: if a registry pod exists, wait for it to be Ready, but do not fail if absent.
echo "[$(ts)] Checking for registry pod(s) for label olm.catalogSource=${catsrc} in ns ${ns}..."
local names
names="$(oc -n "$ns" get pods -l "olm.catalogSource=${catsrc}" -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [[ -n "$names" ]]; then
echo "$names" | sed 's/^/ - /'
echo "[$(ts)] Waiting for pod(s) to be Ready (timeout ${POD_READY_TIMEOUT}s)..."
oc -n "$ns" wait pod -l "olm.catalogSource=${catsrc}" --for=condition=Ready --timeout="${POD_READY_TIMEOUT}s" || true
else
echo "[$(ts)] No registry pod detected yet. Endpoints and READY state are considered sufficient, continuing."
fi
}
get_job_uid() { local name="$1" ns="$2"; oc get job "$name" -n "$ns" -o jsonpath='{.metadata.uid}' 2>/dev/null || true; }
wait_for_new_job() {
local name="$1" ns="$2" old_uid="$3" timeout_secs="$4"
local elapsed=0 step=5
local ref="job/${name} -n ${ns}"
while true; do
local uid; uid="$(get_job_uid "$name" "$ns")"
if [[ -n "$uid" && "$uid" != "$old_uid" ]]; then
echo "[$(ts)] ${ref} is present with UID: ${uid}"
return 0
fi
if (( elapsed >= timeout_secs )); then
echo "[$(ts)] Timed out waiting for ${ref} within ${timeout_secs}s."
return 1
fi
echo "[$(ts)] Waiting for ${ref}... (${elapsed}/${timeout_secs}s)"
sleep "$step"; (( elapsed += step ))
done
}
wait_for_job_success() {
local job_name=$1 ns=$2 timeout=${3:-12000} interval=5
echo "[$(ts)] Waiting for Job '$job_name' in ns '$ns' to succeed (timeout: $timeout sec)..."
SECONDS=0
while true; do
local status; status=$(oc get job "$job_name" -n "$ns" -o jsonpath='{.status.succeeded}' 2>/dev/null)
if [[ "$status" == "1" ]]; then
echo "[$(ts)] Job '$job_name' succeeded."
return 0
fi
local failed; failed=$(oc get job "$job_name" -n "$ns" -o jsonpath='{.status.failed}' 2>/dev/null)
if [[ -n "$failed" && "$failed" -gt 0 ]]; then
echo "[$(ts)] Job '$job_name' failed."
return 1
fi
if (( SECONDS >= timeout )); then
echo "[$(ts)] Timeout waiting for Job '$job_name'."
return 2
fi
sleep $interval
done
}
# -----------------------------
# Validations
# -----------------------------
require oc
# -----------------------------
# Section 0: Backups
# -----------------------------
BACKUP_DIR="${BACKUP_DIR:-./wxo-hotfix-backup-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$BACKUP_DIR"
if oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" >/dev/null 2>&1; then
oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o yaml > "${BACKUP_DIR}/subscription.yaml"
fi
if oc -n "$PROJECT_CPD_INST_OPERATORS" get csv "${CSV_NAME}" >/dev/null 2>&1; then
oc -n "$PROJECT_CPD_INST_OPERATORS" get csv "${CSV_NAME}" -o yaml > "${BACKUP_DIR}/csv.yaml"
fi
if oc -n "$PROJECT_CPD_INST_OPERATORS" get catsrc "$CATSRC_NAME" >/dev/null 2>&1; then
oc -n "$PROJECT_CPD_INST_OPERATORS" get catsrc "$CATSRC_NAME" -o yaml > "${BACKUP_DIR}/catsrc.yaml"
fi
# -----------------------------
# Section 1: Update CatalogSource image and reload, then wait for READY + endpoints
# -----------------------------
if oc -n "$PROJECT_CPD_INST_OPERATORS" get catsrc "$CATSRC_NAME" >/dev/null 2>&1; then
echo "[$(ts)] Patching CatalogSource ${CATSRC_NAME} image to ${CATSRC_IMAGE_VALUE}"
oc -n "$PROJECT_CPD_INST_OPERATORS" patch catsrc "$CATSRC_NAME" \
--type='json' \
--patch "[{\"op\":\"replace\",\"path\":\"/spec/image\",\"value\":\"$CATSRC_IMAGE_VALUE\"}]"
echo "[$(ts)] Restarting catalog pod to pull new image"
oc -n "$PROJECT_CPD_INST_OPERATORS" delete pod -l "olm.catalogSource=${CATSRC_NAME}" --ignore-not-found
# Wait for OLM READY and registry service endpoints
wait_for_catalog_ready "$PROJECT_CPD_INST_OPERATORS" "$CATSRC_NAME" "$CATSRC_READY_TIMEOUT" "$SVC_ENDPOINTS_TIMEOUT"
else
echo "[$(ts)] CatalogSource ${CATSRC_NAME} not found in ${PROJECT_CPD_INST_OPERATORS}, skipping catalog update."
fi
# -----------------------------
# Section 2: Patch CSV with hotfix images
# -----------------------------
patches=(
"0 ${WO_OPERATOR_HOTFIX_IMAGE_VALUE}"
"3 ${DOC_PROC_OPERATOR_HOTFIX_IMAGE_VALUE}"
"5 ${COMPONENT_OPERATOR_HOTFIX_IMAGE_VALUE}"
"1 ${BA_UAB_WF_OPERATOR_HOTFIX_IMAGE_VALUE}"
"2 ${DE_EMPLOYEE_SERVER_HOTFIX_IMAGE_VALUE}"
)
for patch in "${patches[@]}"; do
IFS=' ' read -r index image <<< "$patch"
echo "[$(ts)] Patching CSV ${CSV_NAME} deployment index ${index} with image ${image}"
oc patch csv "$CSV_NAME" --namespace "$PROJECT_CPD_INST_OPERATORS" --type='json' \
--patch="[{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/$index/spec/template/spec/containers/0/image\", \"value\": \"$image\"}]"
done
# Update ADS image envs inside deployment index 4
oc patch csv "$CSV_NAME" --namespace "$PROJECT_CPD_INST_OPERATORS" --type='json' --patch="[ \
{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/4/spec/template/spec/containers/0/env/8/value\", \"value\": \"$ADS_RESTAPI_IMAGE_VALUE\"}, \
{\"op\": \"replace\", \"path\": \"/spec/install/spec/deployments/4/spec/template/spec/containers/0/env/9/value\", \"value\": \"$ADS_PARSING_IMAGE_VALUE\"} \
]"
wait_for_csv_succeeded "$CSV_NAME" "$PROJECT_CPD_INST_OPERATORS" "$CSV_WAIT_TIMEOUT"
# -----------------------------
# Section 2.5: Label the WO CR with hotfix=2
# -----------------------------
WO_CR_NAME="$(oc get wo -n "$PROJECT_CPD_INST_OPERANDS" -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)"
if [[ -n "$WO_CR_NAME" ]]; then
echo "[$(ts)] Adding label hotfix=2 to WO CR $WO_CR_NAME in ns $PROJECT_CPD_INST_OPERANDS"
oc -n "$PROJECT_CPD_INST_OPERANDS" label wo "$WO_CR_NAME" hotfix=2 --overwrite
else
echo "[$(ts)] No WO CR found in namespace $PROJECT_CPD_INST_OPERANDS, skipping label."
fi
# -----------------------------
# Section 3: Recreate and wait for DB schema job
# -----------------------------
old_uid="$(get_job_uid "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" || true)"
job="$(oc get jobs -n "$PROJECT_CPD_INST_OPERANDS" -o name | grep '^job.batch/wo-archer-server-db-schema' || true)"
[[ -n "$job" ]] && oc delete -n "$PROJECT_CPD_INST_OPERANDS" $job --ignore-not-found || true
pods="$(oc get pods -n "$PROJECT_CPD_INST_OPERANDS" -o name | grep '^pod/wo-archer-server-db-schema' || true)"
[[ -n "$pods" ]] && oc delete -n "$PROJECT_CPD_INST_OPERANDS" $pods --ignore-not-found || true
if wait_for_new_job "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" "${old_uid:-}" "$CM_RECREATE_TIMEOUT"; then
wait_for_job_success "$JOB_NAME" "$PROJECT_CPD_INST_OPERANDS" "$JOB_TIMEOUT"
else
echo "[$(ts)] Job ${JOB_NAME} did not appear in time. Manual follow-up required."
exit 1
fi
# -----------------------------
# Section 4 - Restart uiproxy deployment after patching
# -----------------------------
oc rollout restart deployment wo-uiproxy -n "$PROJECT_CPD_INST_OPERANDS"
oc rollout status deployment wo-uiproxy -n "$PROJECT_CPD_INST_OPERANDS" --watch
# -----------------------------
# Final message
# -----------------------------
echo "------------------------------------------------------------------"
echo "[$(ts)] 5.2.1 Hotfix 2 steps completed."
echo "Backups saved under ${BACKUP_DIR}"
echo "Monitor the WO CR status by running:"
echo " oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o yaml | grep -E 'watsonxOrchestrateStatus|hotfix'"
echo "Ensure the WO CR status is 'Completed' and label hotfix=2 is present."
echo "------------------------------------------------------------------"
Make the script executable:
chmod 775 hotfix521.sh
Run the script:
nohup sh hotfix521.sh &
Watch progress:
tail -f nohup.out
Verify CR status and label:
oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o yaml | grep hotfix
[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSVAUS","label":"IBM watsonx Orchestrate Cartridge for IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8mKe0000008OVAIA2","label":"Operate-\u003EInstall"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":""}]
Was this topic helpful?
Document Information
Modified date:
06 October 2025
UID
ibm17247201