Table of Contents

• Executive Summary
• Pre-Upgrade Assessment
• Migration Procedure For Category A or B Environments
• Migration Procedure For Category C Environments
• Recovery Procedure for Unprepared Upgrades
• Frequently Asked Questions
• Support Resources
• Pre-Upgrade Checklist - Prior to initiating upgrade to CPD 5.3.1

Executive Summary

Clients upgrading to Cloud Pak for Data (CPD) version 5.3.1 from environments that were initially deployed with CPD 4.6.5 or earlier may encounter compatibility issues with Global Search indexes. This issue arises because OpenSearch 3.2.0, which is included in CPD 5.3.1, cannot process indexes that were created using Elasticsearch 7.10.2.

Important: This issue is non-destructive. OpenSearch will not initialize if legacy indexes are detected, thereby preventing data corruption. All data remains intact and can be migrated using the procedures outlined in this document. Moreover - data store in Global Search is fully derived. Any data stored in Global Search can be re-derived using bulk syncrhonization from sources.

Affected Environments

This issue affects environments that meet both of the following criteria:

1. Initial production deployment occurred with CPD version 4.6.5 or earlier

2. Upgrade target is CPD version 5.3.1

Environments that were initially deployed with CPD version 4.7.0 or later are not affected and may proceed with standard upgrade procedures.

 Pre-Upgrade Assessment

NOTE: Before starting upgrade also refer to the Pre-Upgrade Checklist section.

Identifying Legacy Indexes

Prior to initiating the upgrade process, execute the Index Version Check script to identify any legacy indexes in your environment.

Index Version Check Script

Save the following script as index_version_check.sh and execute it:

#!/bin/bash

# Simplified Index Version Check Script
# Usage: ./index_version_check.sh
# 
# This script checks the Elasticsearch/OpenSearch version for all indexes
# in an OpenShift cluster environment.
echo "=== Getting Elasticsearch Password from OpenShift ==="

# Try elasticsearch-master-client-secret first, fall back to elasticsearch-master-secret
ES_PASS=$(oc get secret elasticsearch-master-client-secret -ojson 2>/dev/null | jq .data.password -r | base64 -d 2>/dev/null)

if [ -z "$ES_PASS" ]; then
 echo "elasticsearch-master-client-secret not found, trying elasticsearch-master-secret..."
 ES_PASS=$(oc get secret elasticsearch-master-secret -ojson 2>/dev/null | jq .data.password -r | base64 -d 2>/dev/null)
fi

if [ -z "$ES_PASS" ]; then
 echo "ERROR: Unable to retrieve Elasticsearch password from OpenShift"
 echo "Tried: elasticsearch-master-client-secret and elasticsearch-master-secret"
 exit 1
fi

# Determine which pod and container to use
echo ""
echo "=== Detecting Elasticsearch Pod ==="

ES_POD=""
ES_CONTAINER="opensearch"

if oc get pod elasticsearch-master-esnodes-000 &>/dev/null; then
 ES_POD="elasticsearch-master-esnodes-000"
 ES_CONTAINER="opensearch"
 echo "Using pod: $ES_POD (container: $ES_CONTAINER)"
else
 echo "elasticsearch-master-esnodes-000 not found, trying elasticsea-0ac3-ib-6fb9-es-server-esnodes-0..."
 if oc get pod elasticsea-0ac3-ib-6fb9-es-server-esnodes-0 &>/dev/null; then
  ES_POD="elasticsea-0ac3-ib-6fb9-es-server-esnodes-0"
  ES_CONTAINER="elasticsearch"
  echo "Using pod: $ES_POD (container: $ES_CONTAINER)"
 fi
fi

if [ -z "$ES_POD" ]; then
 echo "ERROR: Unable to find Elasticsearch pod"
 echo "Tried: elasticsearch-master-esnodes-000 and elasticsea-0ac3-ib-6fb9-es-server-esnodes-0"
 exit 1
fi


# Base curl command for ES queries via oc exec
ES_CURL="oc exec $ES_POD -c $ES_CONTAINER -- curl -k -s -u elastic:${ES_PASS} -XGET --header 'content-type: application/json'"
ES_URL="https://localhost:9200"


echo ""
echo "=== Fetching All Index Settings ==="

ALL_SETTINGS=$(eval "$ES_CURL --url '${ES_URL}/*/_settings'")

if [ -z "$ALL_SETTINGS" ] || [ "$ALL_SETTINGS" == "{}" ]; then
 echo "ERROR: No indexes found or unable to retrieve index settings."
 exit 1
fi


echo "Retrieved settings for all indexes"
echo ""
echo "=== Checking ES Version for Each Index ==="

# Function to convert internal version number to ES/OpenSearch version string
# OpenSearch format: XXYYZZ AA where XX=major, YY=minor, ZZ=revision, AA=alpha/beta (usually 99)
# Example: 2040299 = 2.4.2, 7100299 = 7.10.2
# MASK value from OpenSearch source: 0x08000000 (134217728)
convert_version() {
 local version_num=$1
 if [ "$version_num" == "null" ] || [ -z "$version_num" ]; then
  echo "unknown"
  return
 fi


 # MASK constant from OpenSearch Version.java
 local MASK=134217728 # 0x08000000

 # Check if this is a masked version (OpenSearch) or legacy (Elasticsearch)
 if [ $version_num -ge $MASK ]; then
  # OpenSearch version - XOR with MASK to get actual version
  version_num=$((version_num ^ MASK))
 fi

 # Extract version components based on OpenSearch/ES format
 # Format: XXYYZZ AA (XX=major, YY=minor, ZZ=revision, AA=build/alpha indicator)
 local major=$((version_num / 1000000))
 local minor=$(((version_num / 10000) % 100))
 local revision=$(((version_num / 100) % 100))
 local build=$((version_num % 100))

 # Return version string (build number typically 99 for releases, not displayed)
 echo "${major}.${minor}.${revision}"
}

# Extract index names from the settings response
INDEXES=$(echo "$ALL_SETTINGS" | jq -r 'keys[]')

# Loop through each index and extract its version from the already-fetched settings
for INDEX in $INDEXES; do
 echo ""
 echo "Index: $INDEX"
 echo "---"


 # Extract version info from the already-fetched settings
 VERSION_NUM=$(echo "$ALL_SETTINGS" | jq -r ".\"${INDEX}\".settings.index.version.created // \"null\"")
 VERSION_STRING=$(echo "$ALL_SETTINGS" | jq -r ".\"${INDEX}\".settings.index.version.created_string // \"null\"")

 if [ "$VERSION_STRING" != "null" ]; then
  echo " ES Version (when created): $VERSION_STRING"
 elif [ "$VERSION_NUM" != "null" ]; then
  READABLE_VERSION=$(convert_version $VERSION_NUM)
  echo " ES Version (when created): $READABLE_VERSION (internal: $VERSION_NUM)"
 else
  echo " ES Version (when created): Could not determine"
 fi
done

echo ""
echo "=== Index Version Check Complete ==="
echo ""
echo "IMPORTANT: If you see indexes with version 7.x.x, these are legacy Elasticsearch"
echo "indexes that are incompatible with OpenSearch 3.2.0 (CPD 5.3.1)."
echo ""
echo "Next steps:"
echo " - If no 7.x indexes found: Proceed with standard upgrade"
echo " - If 7.x indexes found with <100GB data: Plan for index deletion and bulk sync"
echo " - If 7.x indexes found with >=100GB data: Configure OpenSearch 2.19.x compatibility"
echo ""
echo "For detailed guidance, refer to the Known Issue documentation."

Usage:

bash
chmod +x index_version_check.sh
./index_version_check.sh

Determining Migration Category

Based on the script output and your Global Search data volume, identify the appropriate migration category:

Migration Procedures by Category

Migration Procedure For Category A or B Environments

Step 1: Schedule Maintenance Window

Coordinate a maintenance window of 30-60 minutes during which Global Search services will be unavailable. Ensure that current backups have been verified prior to proceeding.

Step 2: Remove Legacy Indexes

Execute the appropriate command based on your source CPD version:

For upgrades from CPD 5.1.3 or earlier:

oc exec elasticsearch-0ac3-ib-6fb9-es-server-esnodes-0 -c elasticsearch -- curl -XDELETE \
--url "http://localhost:19200/<INDEX_NAME>?pretty" \
--header 'content-type: application/json'

For upgrades from CPD 5.2.0 or later:

ES_PASS=$(oc get secret elasticsearch-master-client-secret -ojson | jq .data.password -r | base64 -d)
oc exec elasticsearch-master-esnodes-0 -c opensearch -- curl -k -s -u elastic:$ES_PASS -XDELETE \
--url "https://localhost:9200/<INDEX_NAME>?pretty" \
--header 'content-type: application/json'

Replace <INDEX_NAME> with each legacy index identified during the pre-upgrade assessment.

Step 3: Restart the wkc-search pod

This operation ensures that indexes are regenerated using the compatible OpenSearch version.

oc delete pod -l component=wkc-search

Step 4: Execute Bulk Synchronization Operations

Perform bulk synchronization for all affected data types, please choose appropriate release version (default is 5.2.0):

• Bulk Synchronization of CAMS Assets](https://www.ibm.com/docs/en/software-hub/5.2.x?topic=administering-bulk-sync-assets-global-search)
• Bulk Synchronization of CAMS Relationships
• Bulk Synchronization of Business Glossary 

Step 5: Verification

Validate that all data is accessible through Global Search and that search functionality operates as expected.

Step 6: Complete CPD 5.3.1 Upgrade

Execute the standard CPD 5.3.1 upgrade procedures as documented in the official upgrade guide.

Migration Procedure For Category C Environments

Step 1: Identify Source CPD Version

Determine your current CPD version and corresponding OpenSearch version:

Step 2: Configure CCS Custom Resource (If Required)

For upgrades from CPD 5.2.1 or earlier, add the following properties to your CCS Custom Resource prior to initiating the upgrade:

# Specify the version corresponding to your source CPD version
opensearch_legacy_core_version: "2.19.2" # Example: CPD 5.2.1
opensearch_legacy_plugin_version: "2.19.2.0" # Example: CPD 5.2.1

Adjust the version numbers according to the table in Step 1.

Step 3: Execute CPD 5.3.1 Upgrade

Proceed with standard CPD 5.3.1 upgrade procedures. OpenSearch will continue operating on version 2.19.x, maintaining compatibility with existing legacy indexes.

Note: Migration to OpenSearch 3.2.0 will require a future maintenance window. Contact IBM Support for guidance on planning this migration when your organization is ready to proceed.

Recovery Procedure for Unprepared Upgrades

Recovery Steps

If an upgrade to CPD 5.3.1 was initiated without proper preparation, the following recovery procedure may be executed. Note that the upgrade process is non-destructive; OpenSearch 3.2.0 will not initialize if legacy indexes are detected, preventing data corruption.

Step 1: Enable Maintenance Mode

oc patch ccs ccs-cr --type merge \
--patch '{"spec": {"ignoreForMaintenance": true}}' \
-n ${CPD-INSTANCE]

Step 2: Quiesce OpenSearch Cluster

oc patch clusters.opensearch.cloudpackopen.ibm.com elasticsearch-master \
--type merge \
--patch '{"spec": {"quiesce": true}}'

Step 3: Configure Legacy OpenSearch Version

Update the CCS Custom Resource with the appropriate version corresponding to your source CPD version (refer to the version table in Category C, Step 1):

opensearch_legacy_core_version: "2.19.3" # Adjust based on source version
opensearch_legacy_plugin_version: "2.19.3.0" # Adjust based on source version

Step 4: Disable Maintenance Mode

oc patch ccs ccs-cr --type merge \
--patch '{"spec": {"ignoreForMaintenance": false}}' \
-n cpd-instance

Step 5: Monitor Reconciliation

Monitor CCS operator logs to verify that the cluster stabilizes with the legacy OpenSearch version. Subsequently, proceed with the appropriate migration procedure (Category B or C) based on your data volume.

Frequently Asked Questions

Q: Is data loss possible if the upgrade proceeds without legacy index assessment?

A: No. OpenSearch 3.2.0 will not initialize if legacy indexes are detected, thereby preventing any data corruption. The recovery procedure documented above may be executed to restore service. Moreover - data store in Global Search is fully derived. Any data stored in Global Search can be re-derived using bulk syncrhonization from sources.

Q: What is the expected duration for migration procedures?

A:

• Category B (< 100GB): Approximately 30-60 minutes
• Category C (≥ 100GB): Duration varies significantly based on data volume. Contact IBM Support to discuss migration planning.

Q: Is it permissible to maintain OpenSearch 2.19.x indefinitely?

A: While the current solution allows continued operation on OpenSearch 2.19.x, migration to OpenSearch 3.2.0 will eventually be required. The Category C approach enables CPD 5.3.1 upgrade while allowing proper planning for index migration.

Q: How should category determination be approached if uncertain?

A: Execute the Index Version Check script and assess your Global Search data volume. If uncertainty remains, contact IBM Support for guidance in determining the appropriate migration category.

Q: How should migration to OpenSearch 3.2.0 be planned for Category C environments?

A: Contact IBM Support to discuss migration planning. The migration will require an extended maintenance window, and IBM Support can provide guidance on the recommended approach based on your specific environment and data volume.

Support Resources

Obtaining Assistance - To request support:

1. Execute the Index Version Check script and document results

2. Determine Global Search data volume

3. Contact IBM Support with the collected information

4. Reference this Known Issue document

Pre-Upgrade Checklist - Prior to initiating upgrade to CPD 5.3.1:

☐  Execute Index Version Check script

☐  Determine applicable migration category (A, B, or C)

☐  Review appropriate migration procedures

☐  Verify backup currency and integrity

☐  Configure CCS Custom Resource if required (Category C from CPD 5.2.1 or earlier)