IBM Support

Applying the watsonx Orchestrate 5.2.2 hotfix (Hotfix 1)

Fix Readme


Abstract

Apply this hotfix after completing the watsonx Orchestrate installation for Version 5.2.2, or after upgrading from a previous release to Version 5.2.2.

It takes backup of the CatalogSource, Subscription, and the CSV. It updates the CatalogSource, deletes the CSV and Subscription, and reapplies the backed-up Subscription YAML, allowing the OLM to recreate the CSV from the updated catalog.

Content

1) Set the operator and operand namespaces

export PROJECT_CPD_INST_OPERATORS=<enter your IBM Software Hub operator project>
export PROJECT_CPD_INST_OPERANDS=<enter your IBM Software Hub operand project>
 

2) Verify watsonx Orchestrate CR status

Confirm that the watsonx Orchestrate custom resource in the operands namespace shows Completed status before proceeding.

oc get wo -n "${PROJECT_CPD_INST_OPERANDS}" -o jsonpath='{.items[0].status.watsonxOrchestrateStatus}{"\n"}'
 

3) Back up the CatalogSource, Subscription, and CSV

# CatalogSource
oc get catsrc -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
| xargs -r -I {} oc get catsrc {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
> watson-orchestrate-catsrc-bkup.yaml

# Subscription
oc get subscriptions.operators.coreos.com -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
| xargs -r -I {} oc get subscriptions.operators.coreos.com {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
> watson-orchestrate-subscription-bkup.yaml

# CSV
oc get csv -n "${PROJECT_CPD_INST_OPERATORS}" | grep -i orchestrate | awk '{print $1}' \
| xargs -r -I {} oc get csv {} -n "${PROJECT_CPD_INST_OPERATORS}" -o yaml \
> watson-orchestrate-csv-bkup.yaml
 

4) Mirror images to a private Docker registry if air gapped

Install skopeo: https://github.com/containers/skopeo/blob/main/install.md

export LOCAL_REGISTRY="your_local_registry"
export LOCAL_USER="your_local_registry_username"
export LOCAL_PASS="your_local_registry_password"
export IBM_ENTITLEMENT_KEY="your_ibm_entitlement_key"
export AUTH_JSON_PATH="${HOME}/.airgap/auth.json"

mkdir -p "$(dirname "${AUTH_JSON_PATH}")"

skopeo login cp.icr.io --username cp --password "${IBM_ENTITLEMENT_KEY}" --authfile "${AUTH_JSON_PATH}"
skopeo login "${LOCAL_REGISTRY}" --username "${LOCAL_USER}" --password "${LOCAL_PASS}" --authfile "${AUTH_JSON_PATH}"

Copy operator images
# ibm-watsonx-orchestrate-operator-catalog
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:491e99116ea465e8b1bf46c4ca944e4a27668f810c6802a0a00d3e9e3ef9fcfc \
  docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:491e99116ea465e8b1bf46c4ca944e4a27668f810c6802a0a00d3e9e3ef9fcfc

# ibm-watsonx-orchestrate-operator-bundle
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:aa4ce42d33670ab991d0af6792034739be4f137ee69ed12b5327293de51a3545 \
  docker://$LOCAL_REGISTRY/cpopen/ibm-watsonx-orchestrate-operator-bundle@sha256:aa4ce42d33670ab991d0af6792034739be4f137ee69ed12b5327293de51a3545

# ibm-wxo-component-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-wxo-component-operator@sha256:24c270b317898ea3d5c3cc7affe854b8fa51d75f4ea3f15d20cf07df208eab3d \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-wxo-component-operator@sha256:24c270b317898ea3d5c3cc7affe854b8fa51d75f4ea3f15d20cf07df208eab3d

# ibm-watsonx-orchestrate-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-watsonx-orchestrate-operator@sha256:47a2dbfce423fcc42d46a4f5436ed8b271eae7e97975224735ae79dd703083c7 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-watsonx-orchestrate-operator@sha256:47a2dbfce423fcc42d46a4f5436ed8b271eae7e97975224735ae79dd703083c7

# ibm-document-processing-operator
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://icr.io/cpopen/ibm-document-processing-operator@sha256:f383ea03f75fd8bcb5d03e9753a5276c840525790805541ffefe34ac50f5fec0 \
  docker://${LOCAL_REGISTRY}/cpopen/ibm-document-processing-operator@sha256:f383ea03f75fd8bcb5d03e9753a5276c840525790805541ffefe34ac50f5fec0

Copy operand images
# tools-runtime
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime@sha256:ec0b99a38571e81f820a45f5f3b6302ad384e3beea93e93f1d9b49149bd934e3 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime@sha256:ec0b99a38571e81f820a45f5f3b6302ad384e3beea93e93f1d9b49149bd934e3

# tools-runtime-manager
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-manager@sha256:3104072dc92fd79c35e74e0b09a2007b3d07be48eb465b25fc38a199bc32d952 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-manager@sha256:3104072dc92fd79c35e74e0b09a2007b3d07be48eb465b25fc38a199bc32d952

# tools-runtime-scheduler
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:721322e09cb9e2ceeb33ebab5004f4a292d978ace6abc07feb047e53430dddc2 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/tools-runtime-scheduler@sha256:721322e09cb9e2ceeb33ebab5004f4a292d978ace6abc07feb047e53430dddc2

# wxo-server-conversation_controller
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:91cbfef96e4e65b14845d4e138a158ce2541368460ba3d248507a433af5fd878 \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-conversation_controller@sha256:91cbfef96e4e65b14845d4e138a158ce2541368460ba3d248507a433af5fd878

# wxo-server-server
skopeo copy --all --authfile "${AUTH_JSON_PATH}" --dest-tls-verify=false --src-tls-verify=false \
  docker://cp.icr.io/cp/watsonx-orchestrate/wxo-server-server@sha256:38834d8024f96d44776086655de8f2b676b714b4907f533a5efd7a785feb7bfd \
  docker://${LOCAL_REGISTRY}/cp/watsonx-orchestrate/wxo-server-server@sha256:38834d8024f96d44776086655de8f2b676b714b4907f533a5efd7a785feb7bfd
 

After you copied, verify the images exist in $LOCAL_REGISTRY.

5) Apply the 5.2.2 Hotfix

Create hotfix5221.sh with the following contents: 

#!/bin/sh
# -----------------------------------------------------------------------------
# watsonx Orchestrate 5.2.2 Hotfix
# - Verifies watsonx Orchestrate version from .status.versionStatus.status
# - Backs up Subscription, matching CSVs (name contains ibm-watsonx-orchestrate-operator),
#   and CatalogSource, then EXIT if either Subscription or CatalogSource was not present
# - Deletes CatalogSource first, then Subscription, CSVs, bootstrap Job+same-named CM,
#   and any InstallPlans that reference ibm-watsonx-orchestrate-operator
# - Recreates CatalogSource with the provided image and waits for READY + endpoints
# - Reapplies cleaned Subscription (removes status, resourceVersion, uid, generation,
#   creationTimestamp, kubectl last-applied annotation)
# - Waits for InstallPlan to be approved if required, then proceeds automatically once Complete
# - Labels the WO CR with a configurable label key and value
#   * HOTFIX_LABEL_VALUE for hotfix 0 is 5.2.2.1
#   * If an existing label value matches x.x.x.x and is higher than HOTFIX_LABEL_VALUE,
#     the script exits early after informing you
# - Deletes a fixed set of Jobs in the operands namespace and waits for all to reappear with
#   new UIDs and succeed
# -----------------------------------------------------------------------------

set -eu

# -----------------------------
# Configurable timeouts (seconds)
# -----------------------------
CSV_WAIT_TIMEOUT="${CSV_WAIT_TIMEOUT:-900}"
CM_RECREATE_TIMEOUT="${CM_RECREATE_TIMEOUT:-12000}"
JOB_TIMEOUT="${JOB_TIMEOUT:-12000}"
CATSRC_READY_TIMEOUT="${CATSRC_READY_TIMEOUT:-900}"
SVC_ENDPOINTS_TIMEOUT="${SVC_ENDPOINTS_TIMEOUT:-600}"
POD_READY_TIMEOUT="${POD_READY_TIMEOUT:-600}"
SUB_READY_TIMEOUT="${SUB_READY_TIMEOUT:-900}"
IP_APPROVAL_TIMEOUT="${IP_APPROVAL_TIMEOUT:-900}"
SUB_DELETE_TIMEOUT="${SUB_DELETE_TIMEOUT:-600}"
CSV_DELETE_TIMEOUT="${CSV_DELETE_TIMEOUT:-600}"
JOB_DELETE_TIMEOUT="${JOB_DELETE_TIMEOUT:-300}"
CM_DELETE_TIMEOUT="${CM_DELETE_TIMEOUT:-120}"
IP_DELETE_TIMEOUT="${IP_DELETE_TIMEOUT:-300}"
CATSRC_DELETE_TIMEOUT="${CATSRC_DELETE_TIMEOUT:-300}"

# -----------------------------
# Required namespaces
# -----------------------------
: "${PROJECT_CPD_INST_OPERATORS:?Set PROJECT_CPD_INST_OPERATORS}"
: "${PROJECT_CPD_INST_OPERANDS:?Set PROJECT_CPD_INST_OPERANDS}"

# -----------------------------
# OLM resource names and images
# -----------------------------
SUB_NAME="${SUB_NAME:-ibm-cpd-watsonx-orchestrate-operator}"
CATSRC_NAME="${CATSRC_NAME:-ibm-watsonx-orchestrate-catalog}"
CATSRC_IMAGE_VALUE="${CATSRC_IMAGE_VALUE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator-catalog@sha256:491e99116ea465e8b1bf46c4ca944e4a27668f810c6802a0a00d3e9e3ef9fcfc}"

# Operators-namespace bootstrap job image base to match (tag or digest allowed)
BUNDLE_IMAGE_BASE="${BUNDLE_IMAGE_BASE:-icr.io/cpopen/ibm-watsonx-orchestrate-operator-bundle}"

# Target jobs (single variable, newline separated)
TARGET_JOBS="${TARGET_JOBS:-wo-archer-server-db-schema-job
wo-watson-orchestrate-skill-creation-job-agents-tools
wo-watson-orchestrate-skill-creation-job-applications
wo-watson-orchestrate-skill-creation-job-flow
wo-watson-orchestrate-skill-creation-job-partner-others
wo-watson-orchestrate-skill-creation-job-partner-salesloft
wo-watson-orchestrate-skill-creation-job-partner-workfront
wo-watson-orchestrate-skill-creation-job-productivity}"

# Required WO version
REQUIRED_WO_VERSION="${REQUIRED_WO_VERSION:-5.2.2}"

# CSV prefix to scope IP checks strictly to Orchestrate
WXO_CSV_PREFIX="${WXO_CSV_PREFIX:-ibm-watsonx-orchestrate-operator}"

# Hotfix label configuration
HOTFIX_LABEL_KEY="${HOTFIX_LABEL_KEY:-hotfix}"
HOTFIX_LABEL_VALUE="${HOTFIX_LABEL_VALUE:-5.2.2.1}"

# -----------------------------
# Helpers
# -----------------------------
ts() { date +"%Y-%m-%d %H:%M:%S"; }

require() {
  command -v "$1" >/dev/null 2>&1 || { echo "[$(ts)] Missing required command: $1"; exit 1; }
}

get_wo_version() {
  ns="$1"
  oc get wo -n "$ns" -o jsonpath='{.items[0].status.versionStatus.status}' 2>/dev/null || true
}

is_semver4() {
  v="$1"
  printf '%s' "$v" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'
}

ver_gt() {
  a="$1"; b="$2"
  IFS=. read -r a1 a2 a3 a4 <<EOF
$a
EOF
  IFS=. read -r b1 b2 b3 b4 <<EOF
$b
EOF
  if [ "${a1:-0}" -gt "${b1:-0}" ]; then return 0; fi
  if [ "${a1:-0}" -lt "${b1:-0}" ]; then return 1; fi
  if [ "${a2:-0}" -gt "${b2:-0}" ]; then return 0; fi
  if [ "${a2:-0}" -lt "${b2:-0}" ]; then return 1; fi
  if [ "${a3:-0}" -gt "${b3:-0}" ]; then return 0; fi
  if [ "${a3:-0}" -lt "${b3:-0}" ]; then return 1; fi
  if [ "${a4:-0}" -gt "${b4:-0}" ]; then return 0; fi
  return 1
}

wait_for_csv_succeeded() {
  ns="$1"; csv="$2"; timeout_secs="$3"
  elapsed=0; step=10
  while :; do
    phase="$(oc -n "$ns" get csv "$csv" -o jsonpath='{.status.phase}' 2>/dev/null || true)"
    if [ "$phase" = "Succeeded" ]; then
      echo "[$(ts)] CSV $csv in $ns is Succeeded."
      return 0
    fi
    if [ "$elapsed" -ge "$timeout_secs" ]; then
      echo "[$(ts)] CSV $csv in $ns did not reach Succeeded in ${timeout_secs}s."
      oc -n "$ns" get csv "$csv" -o yaml | sed -n '1,160p' || true
      return 1
    fi
    echo "[$(ts)] Waiting for CSV $csv in $ns... (${elapsed}/${timeout_secs}s)"
    sleep "$step"; elapsed=$((elapsed+step))
  done
}

wait_for_catalog_ready() {
  ns="$1"; catsrc="$2"; cs_timeout="${3:-900}"; svc_timeout="${4:-600}"
  elapsed=0; step=5
  echo "[$(ts)] Waiting for CatalogSource/$catsrc in $ns to reach READY..."
  while :; do
    state="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.connectionState.lastObservedState}' 2>/dev/null || true)"
    msg="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.connectionState.message}' 2>/dev/null || true)"
    if [ -n "$state" ]; then echo "[$(ts)] CatalogSource state: $state ${msg:-}"; fi
    if [ "$state" = "READY" ]; then break; fi
    if [ "$elapsed" -ge "$cs_timeout" ]; then
      echo "[$(ts)] Timeout: CatalogSource did not reach READY in ${cs_timeout}s."
      oc -n "$ns" get catsrc "$catsrc" -o yaml | sed -n '1,200p' || true
      return 1
    fi
    sleep "$step"; elapsed=$((elapsed+step))
  done

  svc_ns="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.registryService.serviceNamespace}' 2>/dev/null || true)"
  svc_name="$(oc -n "$ns" get catsrc "$catsrc" -o jsonpath='{.status.registryService.serviceName}' 2>/dev/null || true)"
  if [ -n "$svc_ns" ] && [ -n "$svc_name" ]; then
    echo "[$(ts)] Waiting for Endpoints/$svc_name in $svc_ns to have addresses..."
    elapsed=0
    while :; do
      cnt="$(oc -n "$svc_ns" get endpoints "$svc_name" -o jsonpath='{range .subsets[*]}{range .addresses[*]}X{end}{end}' 2>/dev/null | wc -c | tr -d ' ')"
      if [ -n "$cnt" ] && [ "$cnt" -ge 1 ]; then
        echo "[$(ts)] Service endpoints are populated."
        break
      fi
      if [ "$elapsed" -ge "$svc_timeout" ]; then
        echo "[$(ts)] Timeout: service endpoints not ready in ${svc_timeout}s."
        oc -n "$svc_ns" get svc "$svc_name" -o yaml | sed -n '1,160p' || true
        oc -n "$svc_ns" get endpoints "$svc_name" -o yaml | sed -n '1,160p' || true
        break
      fi
      sleep "$step"; elapsed=$((elapsed+step))
    done
  fi

  echo "[$(ts)] Checking for registry pod(s) labeled olm.catalogSource=$catsrc..."
  oc -n "$ns" wait pod -l "olm.catalogSource=${catsrc}" --for=condition=Ready --timeout="${POD_READY_TIMEOUT}s" || true
}

wait_for_resource_deleted() {
  ns="$1"; rtype="$2"; rname="$3"; timeout_secs="${4:-300}"
  elapsed=0; step=5
  echo "[$(ts)] Waiting for deletion of ${rtype}/${rname} in $ns (timeout ${timeout_secs}s)"
  if oc -n "$ns" wait --for=delete "${rtype}/${rname}" --timeout="${timeout_secs}s" >/dev/null 2>&1; then
    echo "[$(ts)] Confirmed deleted: ${rtype}/${rname}"
    return 0
  fi
  while :; do
    if ! oc -n "$ns" get "$rtype" "$rname" >/dev/null 2>&1; then
      echo "[$(ts)] Confirmed deleted (via poll): ${rtype}/${rname}"
      return 0
    fi
    if [ "$elapsed" -ge "$timeout_secs" ]; then
      echo "[$(ts)] WARNING: Timed out waiting for ${rtype}/${rname} to delete after ${timeout_secs}s"
      return 1
    fi
    sleep "$step"; elapsed=$((elapsed+step))
  done
}

# Orchestrate IP appearance monitor
wait_for_wxo_ip() {
  ns="$1"; timeout_secs="${2:-900}"; elapsed=0
  while :; do
    line="$(oc -n "$ns" get installplan -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.spec.clusterServiceVersionNames}{"|"}{.status.attachedClusterServiceVersions[*].name}{"|"}{.spec.approved}{"|"}{.status.phase}{"\n"}{end}' 2>/dev/null || true)"
    if printf '%s\n' "$line" | grep -Eq "$WXO_CSV_PREFIX"; then
      echo "[$(ts)] Detected Orchestrate InstallPlan(s):"
      printf '%s\n' "$line" \
      | grep -E "$WXO_CSV_PREFIX" \
      | awk -F'|' 'BEGIN{printf("  %-24s %-32s %-10s %-10s\n","NAME","CSV","APPROVED","PHASE")}
                   {csv=$2; if(length(csv)==0) csv=$3; printf("  %-24s %-32s %-10s %-10s\n",$1,csv,$4,$5)}'
      return 0
    fi
    if [ "$elapsed" -ge "$timeout_secs" ]; then
      echo "[$(ts)] No Orchestrate InstallPlan appeared within ${timeout_secs}s."
      return 1
    fi
    echo "[$(ts)] Waiting for Orchestrate InstallPlan to appear... (${elapsed}/${timeout_secs}s)"
    sleep 5; elapsed=$((elapsed+5))
  done
}

# Prints status and exits as soon as approval is effective
print_manual_approval_waitloop() {
  ns="$1"; sub="$2"; timeout_secs="${3:-900}"
  echo "[$(ts)] Manual approval required. Waiting for approval of Orchestrate InstallPlan..."
  elapsed=0; step=5
  while [ "$elapsed" -lt "$timeout_secs" ]; do
    sub_state="$( oc -n "$ns" get subscriptions.operators.coreos.com "$sub" -o jsonpath='{.status.state}' 2>/dev/null || true )"
    icsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$sub" -o jsonpath='{.status.installedCSV}' 2>/dev/null || true )"
    ccsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$sub" -o jsonpath='{.status.currentCSV}' 2>/dev/null || true )"

    ip_line="$( oc -n "$ns" get installplan -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.spec.clusterServiceVersionNames}{"|"}{.status.attachedClusterServiceVersions[*].name}{"|"}{.spec.approved}{"|"}{.status.phase}{"\n"}{end}' 2>/dev/null || true )"

    if printf '%s\n' "$ip_line" | awk -F'|' '/ibm-watsonx-orchestrate-operator/ && $5=="Complete" {found=1} END{exit(found?0:1)}'; then
      echo "[$(ts)] Detected InstallPlan phase=Complete. Proceeding."
      return 0
    fi

    if [ "$sub_state" = "AtLatestKnown" ] || [ "$sub_state" = "UpgradeSucceeded" ]; then
      echo "[$(ts)] Subscription state=${sub_state}. Proceeding."
      return 0
    fi

    csv_to_check=""
    if [ -n "$icsv" ]; then csv_to_check="$icsv"; elif [ -n "$ccsv" ]; then csv_to_check="$ccsv"; fi
    if [ -n "$csv_to_check" ]; then
      csv_phase="$( oc -n "$ns" get csv "$csv_to_check" -o jsonpath='{.status.phase}' 2>/dev/null || true )"
      if [ "$csv_phase" = "Succeeded" ]; then
        echo "[$(ts)] CSV ${csv_to_check} phase=Succeeded. Proceeding."
        return 0
      fi
    fi

    printf '%s\n' "$ip_line" \
    | grep -E 'ibm-watsonx-orchestrate-operator' \
    | awk -F'|' 'BEGIN{printf("  %-24s %-32s %-10s %-10s\n","NAME","CSV","APPROVED","PHASE")}
                 {csv=$2; if(length(csv)==0) csv=$3; printf("  %-24s %-32s %-10s %-10s\n",$1,csv,$4,$5)}' || true
    echo "[$(ts)] Subscription state: ${sub_state:-unknown}. Still waiting for human approval..."
    sleep "$step"; elapsed=$((elapsed+step))
  done
  echo "[$(ts)] Finished waiting loop. If still pending, please approve the InstallPlan manually in $ns."
  return 0
}

wait_for_subscription_installed_csv() {
  ns="$1"; timeout_secs="${2:-900}"
  elapsed=0
  icsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.installedCSV}' 2>/dev/null || true )"
  ccsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.currentCSV}' 2>/dev/null || true )"
  if [ -n "$icsv" ]; then echo "$icsv"; return 0; fi
  if [ -n "$ccsv" ]; then echo "$ccsv"; return 0; fi
  while :; do
    icsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.installedCSV}' 2>/dev/null || true )"
    ccsv="$( oc -n "$ns" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.currentCSV}' 2>/dev/null || true )"
    if [ -n "$icsv" ]; then echo "$icsv"; return 0; fi
    if [ "$elapsed" -ge "$timeout_secs" ]; then echo ""; return 1; fi
    sleep 5; elapsed=$((elapsed+5))
  done
}

get_job_uid() {
  ns="$1"; name="$2"
  oc -n "$ns" get job "$name" -o jsonpath='{.metadata.uid}' 2>/dev/null || true
}

# -----------------------------
# Validations and version check
# -----------------------------
require oc

echo "[$(ts)] Checking wo.status.versionStatus.status in ${PROJECT_CPD_INST_OPERANDS}"
WO_VER="$(get_wo_version "$PROJECT_CPD_INST_OPERANDS")"
if [ "$WO_VER" != "$REQUIRED_WO_VERSION" ]; then
  echo "[$(ts)] Version check failed. Found '$WO_VER', required '$REQUIRED_WO_VERSION'. Exit."
  exit 1
fi
echo "[$(ts)] Version check passed: $WO_VER"

# -----------------------------
# Early exit if a higher hotfix label already present
# -----------------------------
WO_CR_NAME="$(oc -n "$PROJECT_CPD_INST_OPERANDS" get wo -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)"
if [ -n "$WO_CR_NAME" ]; then
  existing_label="$(oc -n "$PROJECT_CPD_INST_OPERANDS" get wo "$WO_CR_NAME" -o jsonpath="{.metadata.labels.${HOTFIX_LABEL_KEY}}" 2>/dev/null || true)"
  if [ -n "${existing_label:-}" ] && is_semver4 "$existing_label"; then
    if ver_gt "$existing_label" "$HOTFIX_LABEL_VALUE"; then
      echo "[$(ts)] Detected ${HOTFIX_LABEL_KEY}=${existing_label} which is higher than requested ${HOTFIX_LABEL_VALUE}. Exiting without changes."
      exit 0
    fi
  fi
fi

# -----------------------------
# Backups
# -----------------------------
BACKUP_DIR="${BACKUP_DIR:-./wxo-hotfix-backup-$(date +%Y%m%d-%H%M%S)}"
mkdir -p "$BACKUP_DIR"

SUB_PRESENT=0
CATSRC_PRESENT=0

if oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" >/dev/null 2>&1; then
  oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o yaml > "${BACKUP_DIR}/subscription.yaml"
  echo "[$(ts)] Backed up Subscription to ${BACKUP_DIR}/subscription.yaml"
  SUB_PRESENT=1
else
  echo "[$(ts)] Subscription $SUB_NAME not found in $PROJECT_CPD_INST_OPERATORS"
fi

if oc -n "$PROJECT_CPD_INST_OPERATORS" get catsrc "$CATSRC_NAME" >/dev/null 2>&1; then
  oc -n "$PROJECT_CPD_INST_OPERATORS" get catsrc "$CATSRC_NAME" -o yaml > "${BACKUP_DIR}/catsrc.yaml"
  echo "[$(ts)] Backed up CatalogSource to ${BACKUP_DIR}/catsrc.yaml"
  CATSRC_PRESENT=1
else
  echo "[$(ts)] CatalogSource $CATSRC_NAME not found in $PROJECT_CPD_INST_OPERATORS"
fi

CSV_MATCH_LIST="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get csv -o name 2>/dev/null | sed 's#.*/##' | grep -E '^ibm-watsonx-orchestrate-operator' || true)"
if [ -n "$CSV_MATCH_LIST" ]; then
  OUT_CSV="${BACKUP_DIR}/csv-wxo.yaml"
  : > "$OUT_CSV"
  echo "[$(ts)] Backing up matching CSVs to $OUT_CSV"
  echo "$CSV_MATCH_LIST" | while IFS= read CSVN; do
    [ -z "$CSVN" ] && continue
    oc -n "$PROJECT_CPD_INST_OPERATORS" get csv "$CSVN" -o yaml >> "$OUT_CSV" || true
    printf '%s\n' '---' >> "$OUT_CSV"
  done
else
  echo "[$(ts)] No matching CSVs to back up."
fi

if [ "$SUB_PRESENT" -ne 1 ] || [ "$CATSRC_PRESENT" -ne 1 ]; then
  echo "[$(ts)] Required backup missing. Subscription present: $SUB_PRESENT, CatalogSource present: $CATSRC_PRESENT. Exiting."
  exit 1
fi

# -----------------------------
# Delete CatalogSource first
# -----------------------------
echo "[$(ts)] Deleting CatalogSource $CATSRC_NAME in $PROJECT_CPD_INST_OPERATORS"
oc -n "$PROJECT_CPD_INST_OPERATORS" delete catsrc "$CATSRC_NAME" --ignore-not-found || true
wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "catsrc" "$CATSRC_NAME" "$CATSRC_DELETE_TIMEOUT" || true

# -----------------------------
# Delete Subscription and matching CSVs
# -----------------------------
echo "[$(ts)] Deleting Subscription $SUB_NAME in $PROJECT_CPD_INST_OPERATORS"
oc -n "$PROJECT_CPD_INST_OPERATORS" delete subscriptions.operators.coreos.com "$SUB_NAME" --ignore-not-found || true
wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "subscriptions.operators.coreos.com" "$SUB_NAME" "$SUB_DELETE_TIMEOUT" || true

echo "[$(ts)] Deleting CSVs matching '^ibm-watsonx-orchestrate-operator' in $PROJECT_CPD_INST_OPERATORS"
CSV_LIST="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get csv -o name 2>/dev/null | sed 's#.*/##' | grep -E '^ibm-watsonx-orchestrate-operator' || true)"
if [ -n "$CSV_LIST" ]; then
  echo "$CSV_LIST" | sed 's/^/ - /'
  echo "$CSV_LIST" | while IFS= read CSV_NAME; do
    [ -z "$CSV_NAME" ] && continue
    oc -n "$PROJECT_CPD_INST_OPERATORS" delete csv "$CSV_NAME" --ignore-not-found || true
    wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "csv" "$CSV_NAME" "$CSV_DELETE_TIMEOUT" || true
  done
else
  echo "[$(ts)] No matching CSVs found to delete."
fi

# -----------------------------
# Delete operators-namespace bootstrap Job and same-named CM if env CONTAINER_IMAGE starts with bundle base
# -----------------------------
echo "[$(ts)] Searching Jobs in $PROJECT_CPD_INST_OPERATORS with CONTAINER_IMAGE starting with $BUNDLE_IMAGE_BASE"
JOBS_IN_NS="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get jobs -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' 2>/dev/null || true)"
if [ -n "$JOBS_IN_NS" ]; then
  echo "$JOBS_IN_NS" | while IFS= read JN; do
    [ -z "$JN" ] && continue
    envdump="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get job "$JN" -o jsonpath='{range .spec.template.spec.containers[*].env[*]}{.name}:{.value}{"\n"}{end}' 2>/dev/null || true)"
    echo "$envdump" | grep -Eq "^CONTAINER_IMAGE:${BUNDLE_IMAGE_BASE}(:[A-Za-z0-9._-]+|@sha256:[0-9a-f]\{64\})?$" || continue
    echo "[$(ts)] Deleting Job $JN and same-named ConfigMap if present."
    oc -n "$PROJECT_CPD_INST_OPERATORS" delete job "$JN" --ignore-not-found >/dev/null 2>&1 || true
    wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "job" "$JN" "$JOB_DELETE_TIMEOUT" || true
    oc -n "$PROJECT_CPD_INST_OPERATORS" delete cm "$JN" --ignore-not-found >/dev/null 2>&1 || true
    wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "cm" "$JN" "$CM_DELETE_TIMEOUT" || true
  done
else
  echo "[$(ts)] No Jobs found in $PROJECT_CPD_INST_OPERATORS."
fi

# -----------------------------
# Delete InstallPlans that reference Orchestrate CSVs
# -----------------------------
echo "[$(ts)] Deleting InstallPlans referencing $WXO_CSV_PREFIX in $PROJECT_CPD_INST_OPERATORS"
oc -n "$PROJECT_CPD_INST_OPERATORS" get installplan -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.spec.clusterServiceVersionNames}{"|"}{.status.attachedClusterServiceVersions[*].name}{"\n"}{end}' 2>/dev/null \
| while IFS='|' read ip_name csv_specs csv_status; do
    [ -z "$ip_name" ] && continue
    printf '%s\n%s\n' "$csv_specs" "$csv_status" | grep -Eq "$WXO_CSV_PREFIX" || continue
    echo "[$(ts)] Deleting InstallPlan $ip_name"
    oc -n "$PROJECT_CPD_INST_OPERATORS" delete installplan "$ip_name" --ignore-not-found >/dev/null 2>&1 || true
    wait_for_resource_deleted "$PROJECT_CPD_INST_OPERATORS" "installplan" "$ip_name" "$IP_DELETE_TIMEOUT" || true
  done

# -----------------------------
# Prepare cleaned Subscription YAML
# -----------------------------
SUB_BKP="${BACKUP_DIR}/subscription.yaml"
SUB_CLEAN="${BACKUP_DIR}/subscription.clean.yaml"
echo "[$(ts)] Cleaning Subscription backup -> ${SUB_CLEAN}"

if command -v yq >/dev/null 2>&1; then
  yq 'del(.status)
      | del(.metadata.creationTimestamp)
      | del(.metadata.generation)
      | del(.metadata.resourceVersion)
      | del(.metadata.uid)
      | del(.metadata.annotations."kubectl.kubernetes.io/last-applied-configuration")' \
      "$SUB_BKP" > "$SUB_CLEAN"
else
  awk '
    function ltrim(s){ sub(/^[[:space:]]+/,"",s); return s }
    {
      m = match($0, /[^ ]/); indent = m ? m - 1 : 0
      line = $0
      trimmed = ltrim(line)

      if (skip_block && indent <= skip_indent) { skip_block=0 }
      if (skip_status && indent == 0) { skip_status=0 }

      if (skip_block || skip_status) { next }

      if (indent == 0) {
        if (trimmed ~ /^metadata:[[:space:]]*$/) in_meta=1
        else in_meta=0
      }

      if (indent == 0 && trimmed ~ /^status:[[:space:]]*$/) { skip_status=1; next }

      if (in_meta) {
        if (trimmed ~ /^creationTimestamp:[[:space:]]/ \
            || trimmed ~ /^generation:[[:space:]]/ \
            || trimmed ~ /^resourceVersion:[[:space:]]/ \
            || trimmed ~ /^uid:[[:space:]]/) { next }
      }

      if (index(trimmed, "kubectl.kubernetes.io/last-applied-configuration:") == 1) {
        skip_block=1; skip_indent=indent; next
      }

      print $0
    }
  ' "$SUB_BKP" > "$SUB_CLEAN"
fi

# -----------------------------
# Recreate CatalogSource with new image and wait
# -----------------------------
echo "[$(ts)] Recreating CatalogSource $CATSRC_NAME with image $CATSRC_IMAGE_VALUE"

if [ -f "${BACKUP_DIR}/catsrc.yaml" ] && command -v yq >/dev/null 2>&1; then
  yq ".spec.image = \"${CATSRC_IMAGE_VALUE}\"" "${BACKUP_DIR}/catsrc.yaml" | oc -n "$PROJECT_CPD_INST_OPERATORS" apply -f -
else
  cat <<YAML | oc -n "$PROJECT_CPD_INST_OPERATORS" apply -f -
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: ${CATSRC_NAME}
  namespace: ${PROJECT_CPD_INST_OPERATORS}
spec:
  sourceType: grpc
  image: ${CATSRC_IMAGE_VALUE}
  displayName: Watson Orchestrate Catalog
  publisher: IBM
YAML
fi

wait_for_catalog_ready "$PROJECT_CPD_INST_OPERATORS" "$CATSRC_NAME" "$CATSRC_READY_TIMEOUT" "$SVC_ENDPOINTS_TIMEOUT"

# -----------------------------
# Reapply Subscription and monitor IPs without approving
# -----------------------------
echo "[$(ts)] Reapplying Subscription from cleaned backup ${SUB_CLEAN}"

if ! oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" >/dev/null 2>&1; then
  echo "[$(ts)] Subscription not found. Creating..."
  oc create -f "${SUB_CLEAN}"
else
  echo "[$(ts)] Subscription exists. Updating with server-side apply..."
  if ! oc apply --server-side --field-manager=wxo-hotfix -f "${SUB_CLEAN}"; then
    echo "[$(ts)] Server-side apply failed. Forcing replace to avoid resourceVersion errors..."
    oc replace --force -f "${SUB_CLEAN}"
  fi
fi

echo "[$(ts)] Checking for OperatorGroup in ${PROJECT_CPD_INST_OPERATORS}"
oc -n "$PROJECT_CPD_INST_OPERATORS" get operatorgroup -o name || true

if wait_for_wxo_ip "$PROJECT_CPD_INST_OPERATORS" "$IP_APPROVAL_TIMEOUT"; then
  sub_mode="$( oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.spec.installPlanApproval}' 2>/dev/null || echo "" )"
  [ -z "$sub_mode" ] && sub_mode="Automatic"
  if [ "$sub_mode" = "Manual" ]; then
    print_manual_approval_waitloop "$PROJECT_CPD_INST_OPERATORS" "$SUB_NAME" "$IP_APPROVAL_TIMEOUT" || true
  else
    echo "[$(ts)] Subscription approval mode is Automatic. OLM should proceed without manual approval."
  fi
else
  echo "[$(ts)] No Orchestrate InstallPlan appeared before timeout (${IP_APPROVAL_TIMEOUT}s)."
fi

echo "[$(ts)] Snapshot before wait: installedCSV=$(oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.installedCSV}' 2>/dev/null || true), currentCSV=$(oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.currentCSV}' 2>/dev/null || true), state=$(oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.state}' 2>/dev/null || true)"

NEW_CSV="$(wait_for_subscription_installed_csv "$PROJECT_CPD_INST_OPERATORS" "$SUB_READY_TIMEOUT" || echo "")"
echo "[$(ts)] Post-wait sub state: $(oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o jsonpath='{.status.state}' 2>/dev/null || true)"

if [ -z "$NEW_CSV" ]; then
  echo "[$(ts)] Subscription did not report installedCSV/currentCSV in time. Diagnostics follow."
  oc -n "$PROJECT_CPD_INST_OPERATORS" get subscriptions.operators.coreos.com "$SUB_NAME" -o yaml | sed -n '1,180p' || true
  oc -n "$PROJECT_CPD_INST_OPERATORS" get installplan -o wide || true

  CANDIDATE_IP="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get installplan -o jsonpath='{range .items[*]}{.metadata.name}{"|"}{.spec.clusterServiceVersionNames}{"\n"}{end}' 2>/dev/null | awk -F'|' '/orchestrate-operator/ {print $1}' | tail -n1 || true)"
  if [ -n "$CANDIDATE_IP" ]; then
    echo "[$(ts)] Candidate InstallPlan: $CANDIDATE_IP"
    oc -n "$PROJECT_CPD_INST_OPERATORS" describe installplan "$CANDIDATE_IP" | sed -n '1,200p' || true
    echo "[$(ts)] Manual approval may be required. Not approving automatically."
  fi

  if [ -z "$CANDIDATE_IP" ]; then
    echo "[$(ts)] Discovering CSV by label operators.coreos.com/${SUB_NAME}.${PROJECT_CPD_INST_OPERATORS}"
    NEW_CSV="$(oc -n "$PROJECT_CPD_INST_OPERATORS" get csv -l "operators.coreos.com/${SUB_NAME}.${PROJECT_CPD_INST_OPERATORS}" -o name 2>/dev/null | sed 's#.*/##' | grep -E '^ibm-watsonx-orchestrate-operator' | sort | tail -n1 || true)"
    echo "[$(ts)] Discovered CSV candidate: ${NEW_CSV}"
  fi

  if [ -z "$NEW_CSV" ]; then
    echo "[$(ts)] Could not determine CSV automatically. Listing CSVs:"
    oc -n "$PROJECT_CPD_INST_OPERATORS" get csv || true
    echo "[$(ts)] Hints:"
    echo " - If approval mode is Manual, approve the Orchestrate InstallPlan in ${PROJECT_CPD_INST_OPERATORS}."
    echo " - Verify CatalogSource is READY and points to the expected image."
    echo " - Verify an OperatorGroup exists targeting the correct namespaces."
    echo " - Check InstallPlan events for dependency or permission failures."
  fi
fi

if [ -n "$NEW_CSV" ]; then
  echo "[$(ts)] Using CSV ${NEW_CSV}. Waiting for CSV to reach Succeeded..."
  if ! wait_for_csv_succeeded "$PROJECT_CPD_INST_OPERATORS" "$NEW_CSV" "$CSV_WAIT_TIMEOUT"; then
    echo "[$(ts)] WARNING: CSV ${NEW_CSV} did not reach Succeeded within timeout. Install may still be pending approval or reconciling."
  fi
fi

# -----------------------------
# Label WO CR with configurable label and value
# -----------------------------
if [ -n "$WO_CR_NAME" ]; then
  current_label="$(oc -n "$PROJECT_CPD_INST_OPERANDS" get wo "$WO_CR_NAME" -o jsonpath="{.metadata.labels.${HOTFIX_LABEL_KEY}}" 2>/dev/null || true)"
  if [ "$current_label" = "$HOTFIX_LABEL_VALUE" ]; then
    echo "[$(ts)] WO CR ${WO_CR_NAME} already labeled ${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE}"
  else
    echo "[$(ts)] Setting label ${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE} on WO CR ${WO_CR_NAME} in ns ${PROJECT_CPD_INST_OPERANDS}"
    oc -n "$PROJECT_CPD_INST_OPERANDS" label wo "$WO_CR_NAME" "${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE}" --overwrite >/dev/null 2>&1 || true
    new_label="$(oc -n "$PROJECT_CPD_INST_OPERANDS" get wo "$WO_CR_NAME" -o jsonpath="{.metadata.labels.${HOTFIX_LABEL_KEY}}" 2>/dev/null || true)"
    if [ "$new_label" = "$HOTFIX_LABEL_VALUE" ]; then
      echo "[$(ts)] Label set: ${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE}"
    else
      echo "[$(ts)] WARNING: could not confirm ${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE} label was set"
    fi
  fi
else
  echo "[$(ts)] No WO CR found in ns ${PROJECT_CPD_INST_OPERANDS}, skipping label."
fi

# -----------------------------
# Delete fixed Jobs in operands ns, wait for all to reappear with new UIDs, then succeed
# -----------------------------
NS="$PROJECT_CPD_INST_OPERANDS"

to_touch="$(printf '%s\n' "$TARGET_JOBS" | sed '/^$/d' | sort -u)"

if [ -n "$to_touch" ]; then
  uidfile="$(mktemp)"
  printf '%s\n' "$to_touch" | while IFS= read j; do
    olduid="$(get_job_uid "$NS" "$j" || true)"
    printf '%s|%s\n' "$j" "$olduid" >> "$uidfile"
  done

  echo "[$(ts)] Deleting target jobs and their pods if present:"
  printf ' - %s\n' $to_touch
  printf '%s\n' "$to_touch" | while IFS= read j; do
    oc -n "$NS" delete job "$j" --ignore-not-found >/dev/null 2>&1 || true
    oc -n "$NS" delete pod -l "job-name=$j" --ignore-not-found >/dev/null 2>&1 || true
  done

  echo "[$(ts)] Waiting for ALL listed jobs to appear or reappear with new UIDs..."
  pending_reappear="$(printf '%s\n' $to_touch)"
  reappeared_list=""
  start_ts="$(date +%s)"
  step=5
  while [ -n "${pending_reappear:-}" ]; do
    now="$(date +%s)"; elapsed=$(( now - start_ts ))
    if [ "$elapsed" -ge "$CM_RECREATE_TIMEOUT" ]; then
      echo "[$(ts)] ERROR: Reappear timeout reached (${CM_RECREATE_TIMEOUT}s). Still missing:"
      printf ' - %s\n' $pending_reappear
      rm -f "$uidfile"
      exit 1
    fi
    still=""
    for j in $pending_reappear; do
      olduid="$(grep -E "^${j}\|" "$uidfile" | cut -d'|' -f2)"
      uid_now="$(get_job_uid "$NS" "$j" || true)"
      if [ -n "$uid_now" ] && [ "$uid_now" != "$olduid" ]; then
        echo "[$(ts)] Job $j present with new UID."
        reappeared_list="$(printf '%s\n%s\n' "$reappeared_list" "$j" | sed '/^$/d')"
      else
        still="$(printf '%s\n%s\n' "$still" "$j" | sed '/^$/d')"
      fi
    done
    pending_reappear="$still"
    if [ -n "$pending_reappear" ]; then
      echo "[$(ts)] Still waiting to reappear: $(echo "$pending_reappear" | tr '\n' ' ' | sed 's/ $//')"
      sleep "$step"
    fi
  done
  rm -f "$uidfile"

  echo "[$(ts)] Waiting for ALL listed jobs to succeed..."
  pending_success="$(printf '%s\n' "$reappeared_list" | sed '/^$/d')"
  any_fail=0
  start_ts2="$(date +%s)"
  while [ -n "$pending_success" ]; do
    now2="$(date +%s)"; elapsed2=$(( now2 - start_ts2 ))
    if [ "$elapsed2" -ge "$JOB_TIMEOUT" ]; then
      echo "[$(ts)] ERROR: Success timeout reached (${JOB_TIMEOUT}s). Still not succeeded:"
      printf ' - %s\n' $pending_success
      exit 1
    fi

    still2=""
    for j in $pending_success; do
      succ="$(oc -n "$NS" get job "$j" -o jsonpath='{.status.succeeded}' 2>/dev/null || true)"
      failc="$(oc -n "$NS" get job "$j" -o jsonpath='{.status.failed}' 2>/dev/null || true)"
      if [ "$succ" = "1" ]; then
        echo "[$(ts)] Job $j succeeded."
      elif [ -n "$failc" ] && [ "$failc" != "0" ]; then
        echo "[$(ts)] ERROR: Job $j failed."
        any_fail=1
      else
        still2="$(printf '%s\n%s\n' "$still2" "$j" | sed '/^$/d')"
      fi
    done
    pending_success="$still2"
    if [ -n "$pending_success" ]; then
      echo "[$(ts)] Still waiting to succeed: $(echo "$pending_success" | tr '\n' ' ' | sed 's/ $//')"
      sleep 5
    fi
  done

  if [ "${any_fail:-0}" -ne 0 ]; then
    echo "[$(ts)] One or more jobs failed. Check operator logs and events."
    exit 1
  fi
else
  echo "[$(ts)] No jobs listed to process in operands namespace."
fi

# -----------------------------
# Final message
# -----------------------------
echo "------------------------------------------------------------------"
echo "[$(ts)] 5.2.2 Hotfix steps completed."
echo "Backups saved under ${BACKUP_DIR}"
echo "Monitor the watsonx Orchestrate CR status by running:"
echo " oc get wo -n ${PROJECT_CPD_INST_OPERANDS} -o yaml | grep -E 'watsonxOrchestrateStatus|${HOTFIX_LABEL_KEY}'"
echo "Ensure the watsonx Orchestrate CR status is 'Completed' and label ${HOTFIX_LABEL_KEY}=${HOTFIX_LABEL_VALUE} is present."
echo "------------------------------------------------------------------"
 

Make the script executable:

chmod 775 hotfix5221.sh
 

Run the script:

nohup sh hotfix5221.sh &
 

Watch progress:

tail -f nohup.out
 

Verify CR status and label:

oc get wo -n "${PROJECT_CPD_INST_OPERANDS}" -o yaml | grep hotfix
      hotfix: 5.2.2.1

[{"Type":"MASTER","Line of Business":{"code":"LOB76","label":"Data Platform"},"Business Unit":{"code":"BU048","label":"IBM Software"},"Product":{"code":"SSVAUS","label":"IBM watsonx Orchestrate Cartridge for IBM Cloud Pak for Data"},"ARM Category":[{"code":"a8mKe0000008OVAIA2","label":"Operate-\u003EInstall"}],"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":""}]

Document Information

Modified date:
26 November 2025

UID

ibm17250722

Page Feedback