Upgrading watsonx.data intelligence from Version 5.3 to 5.4

An instance administrator can upgrade watsonx.data intelligence from Version 5.3 to Version 5.4.

Who needs to complete this task?

Instance administrator To upgrade watsonx.data intelligence, you must be an instance administrator. An instance administrator has permission to manage software in the following projects:

The operators project for the instance: The operators for this instance of watsonx.data intelligence are installed in the operators project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERATORS} environment variable refers to the operators project.
The operands project for the instance: The custom resources for the control plane and watsonx.data intelligence are installed in the operands project. In the upgrade commands, the ${PROJECT_CPD_INST_OPERANDS} environment variable refers to the operands project.

When do you need to complete this task?

Review the following options to determine whether you need to complete this task:

If you want to upgrade the IBM Software Hub control plane and one or more services at the same time, follow the process in Upgrading an instance of IBM Software Hub instead.
If you didn't upgrade watsonx.data intelligence when you upgraded the IBM Software Hub control plane, complete this task to upgrade watsonx.data intelligence.
Repeat as needed If you are responsible for multiple instances of IBM Software Hub, you can repeat this task to upgrade more instances of watsonx.data intelligence on the cluster.

Information you need to complete this task

Review the following information before you upgrade watsonx.data intelligence:

Version requirements: All the components that are associated with an instance of IBM Software Hub must be installed at the same release. For example, if the IBM Software Hub control plane is at Version 5.4.0, you must upgrade watsonx.data intelligence to Version 5.4.0.

Environment variables

The commands in this task use environment variables so that you can run the commands exactly as written.

If you don't have the script that defines the environment variables, see Setting up installation environment variables.
To use the environment variables from the script, you must source the environment variables before you run the commands in this task. For example, run:
```
source ./cpd_vars.sh
```

Common core services: watsonx.data intelligence requires the IBM Software Hub common core services.
If the common core services are not at the correct version in the operands project for the instance, the common core services are automatically upgraded when you upgrade watsonx.data intelligence. The common core services upgrade increases the amount of time the upgrade takes to complete.

Before you begin

This task assumes that the following prerequisites are met:

System requirements

This task assumes that the cluster meets the minimum requirements for watsonx.data intelligence.

Where to find more information
If this task is not complete, see System requirements.

In addition, if you plan to use features that require GPU, ensure that you have the appropriate type and number of GPU for watsonx.data intelligence.

Where to find more information
If this task is not complete, see GPU requirements.

Workstation

This task assumes that the workstation from which you will run the upgrade is set up as a client workstation and has the following command-line interfaces:

IBM Software Hub CLI: cpd-cli
OpenShift® CLI: oc
Helm CLI: helm

Where to find more information
If this task is not complete, see Updating client workstations.

Control plane

This task assumes that the IBM Software Hub control plane is upgraded.

Where to find more information
If this task is not complete, see Upgrading an instance of IBM Software Hub.

Private container registry

If your environment uses a private container registry (for example, your cluster is air-gapped), this task assumes that the following tasks are complete:

The watsonx.data intelligence software images are mirrored to the private container registry.
Where to find more information

If this task is not complete, see Mirroring images to a private container registry.
The cpd-cli is configured to pull the olm-utils-v4 image from the private container registry.
Where to find more information

If this task is not complete, see Pulling the olm-utils-v4 image from the private container registry.

GPU operators

Where to find more information
If this task is not complete, see Mirroring images to a private container registry.

Where to find more information
If this task is not complete, see Pulling the olm-utils-v4 image from the private container registry.

If you plan to use features that require GPUs, this task assumes that the operators required to use GPUs are installed.

Where to find more information
If this task is not complete, see Installing operators for services that require GPUs.

Red Hat® OpenShift AI

This task assumes that Red Hat OpenShift AI is installed.

Where to find more information
If this task is not complete, see Installing Red Hat OpenShift AI.

Cluster-scoped resources

This task assumes that the cluster-scoped resources, such as custom resource definitions, cluster roles, and cluster role bindings, were updated.

Where to find more information
If this task is not complete, see Updating the cluster-scoped resources for the platform and services.

Procedure

Complete the following tasks to upgrade watsonx.data intelligence:

Specifying installation options
Reverting temporary patches
Upgrading the service
Validating the upgrade
What to do next

Specifying installation options

When you upgrade watsonx.data intelligence, the options that you specified when you installed watsonx.data intelligence are used.

Specify the following options in the install-options.yml file in the cpd-cli work directory (For example: cpd-cli-workspace/olm-utils-workspace/work) only if you want to modify the behavior of watsonx.data intelligence.

The sample YAML content uses the default values.

Retain the --- syntax at the beginning of the entry to ensure that this entry is treated as a separate document.

---
# ............................................................................
# watsonx.data intelligence parameters
# ............................................................................
non_olm:
  watsonxDataintelligence:
    enableAISearch: false
    enableDataGovernanceCatalog: true
    enableKnowledgeGraph: true
    enableDataQuality: false
    enableDataLineage: true
    enableDataProduct: true
    enableUnstructuredDataIntegration: false
    enableGenerativeAICapabilities: true
    enableSemanticEnrichment: true
    enableSemanticEmbedding: false
    enableTextToSql: false
    enableModelsOn: cpu

Property	Description
`enableAISearch`	Specify whether to enable LLM-based semantic search for assets and artifacts across all workspaces. Default value `false` Valid values `false` Do not enable LLM-based semantic search. `true` Enable LLM-based semantic search. If you set `enableAISearch: true`, you must set at least one of the following options to `true`: `enableDataGovernanceCatalog` By default, `enableDataGovernanceCatalog` is set to `true`. `enableDataProduct` By default, `enableDataProduct` is set to `true`.
`enableDataGovernanceCatalog`	Specify whether to enable data governance and catalog features. Default value `true` Valid values `false` Do not enable the data governance and catalog features. `true` Enable the data governance and catalog features. You must set `enableDataGovernanceCatalog: true` if you plan to use the following features: Data quality (`enableDataQuality: true`) Knowledge graph (`enableKnowledgeGraph: true`) Tip: You can enable LLM-based search on the assets and artifacts in the catalog by setting `enableAISearch: true`.
`enableKnowledgeGraph`	Specify whether to enable the knowledge graph feature. The knowledge graph provides the following capabilities: Relationship explorer Business term relationship search Prerequisite This feature requires the data governance catalog. You must set `enableDataGovernanceCatalog:true`. Default value `true` Valid values `false` Do not enable the knowledge graph feature. `true` Enable the knowledge graph feature.
`enableDataQuality`	Specify whether to enable data quality features in projects so that you can measure, monitor, and maintain the quality of your data to ensure the data meets your expectations and standards for specific use cases. Important: When you enable the data quality feature, DataStage Enterprise is automatically installed. If you did not purchase a separate DataStage license, use of DataStage Enterprise is limited to creating, managing, and running data quality rules. For examples of accepted use, see Enabling additional features after installation or upgrade for watsonx.data™ intelligence. Prerequisite This feature requires the data governance catalog. You must set `enableDataGovernanceCatalog:true`. Default value `false` Valid values `false` Do not enable the data quality feature. `true` Enable the data quality feature.
`enableDataLineage`	Specify whether to enable data lineage features. Data lineage is the process of tracking data as it is moved and used by different software tools. Lineage tracks where data came from, how it was transformed, and where the data was moved to. Default value `true` Valid values `false` Do not enable data lineage features. `true` Enable data lineage features.
`enableDataProduct`	Specify whether to enable data sharing features. When you enable data sharing, data producers can package data and data-related assets into data products so that data consumers have access to secure, high quality data Default value `true` Valid values `false` Do not enable data sharing features. `true` Enable data sharing features. Tip: You can enable LLM-based search on data products by setting `enableAISearch: true`.
`enableUnstructuredDataIntegration`	Specify whether to enable unstructured data processing. Install Unstructured Data Integration to ingest, transform, and enrich unstructured data from diverse sources. If you enable unstructured data processing and the data governance and catalog features (`enableDataGovernanceCatalog: true`), you can govern and catalog unstructured data. Prerequisites This feature requires local GPU. If you want to use this feature, you cannot configure watsonx.data intelligence to use remote GPU (`enableModelsOn: 'remote'`). This feature requires at least 3 GPUs. If you want to use entity extraction, you must have 2 additional GPUs. In addition, you must install the following software before you install watsonx.data intelligence: Red Hat OpenShift AI watsonx.data You must also provision a Spark instance. Default value `false` Valid values `false` Do not install Unstructured Data Integration. `true` Install Unstructured Data Integration.
`enableGenerativeAICapabilities`	Specify whether to enable gen AI capabilities. Enable the gen AI capabilities if you plan to use the following features: Semantic enrichment Text to SQL Default value `true` Valid values `false` Do not enable generative AI capabilities. `true` Enable generative AI capabilities.
`enableSemanticEnrichment`	Specify whether to enable gen AI metadata expansion. Metadata expansion includes: Table name expansion Column name expansion Description generation Prerequisite This feature requires gen AI capabilities. You must set `enableGenerativeAICapabilities: true`. Default value `true` Valid values `false` Do not enable gen AI metadata expansion. `true` Enable gen AI metadata expansion.
`enableSemanticEmbedding`	Specify whether to enable semantic embedding. You must enable semantic embedding if you plan to use the following features: Text to SQL Prerequisite This feature requires GPU. You cannot run the required model on CPU. In addition, this feature requires gen AI capabilities. You must set `enableGenerativeAICapabilities: true`. Default value `false` Valid values `false` Do not enable semantic embedding. `true` Enable semantic embedding.
`enableTextToSql`	Specify whether to generate SQL queries from natural language input. Text-to-SQL capabilities can be used to create query-based data assets, which can be use for data products or in searches. Prerequisite This feature requires GPU. You can choose where to run the required models: To run the required models locally, set `enableModelsOn: gpu` To run the required models on a remote instance of watsonx.ai™, set `enableModelsOn: remote` In addition, this feature requires the following settings: Semantic embedding. You must set `enableSemanticEmbedding: true`. Default value `false` Valid values `false` Do not convert natural language queries to SQL queries. `true` Convert natural language queries to SQL queries.
`enableModelsOn`	Specify where you want the models that are used with the gen AI capabilities to run. Prerequisite This feature requires gen AI capabilities. You must set `enableGenerativeAICapabilities: true`. Default value `'cpu'` Valid values `'cpu'` Run the foundation model on CPU. Restriction: This option can be used only for expanding metadata and term assignment when enriching metadata (`enableSemanticEnrichment: true`). This option is not supported for converting natural language queries to SQL queries ( `enableTextToSql: true`). `'gpu'` Run the foundation model on GPU. Important: If you use this setting, the inference foundation models component (`watsonx_ai_ifm`) is automatically installed. This option requires at least one GPU. For information about supported GPUs, see GPU requirements for models. `'remote'` Run the foundation model on a remote instance of watsonx.ai. The instance can be running on: Another on-premises instance of IBM Software Hub IBM watsonx™ as a Service Important: If you use this setting, you must: Ensure that the foundation model is available and running on the remote instance. Create a connection to the remote instance. For more information, see Enabling users to connect to an external IBM watsonx.ai foundation model in the Data Fabric documentation. If the preceding requirements are not met, any tasks that rely on the model will fail.
`customModelTextToSql`	.Deprecated This option is deprecated and will be removed in a future release. Specify a custom model for Text-To-SQL conversions. Default model By default, the Text-To-SQL feature uses the granite-4-h-small model (ID: `granite-4-h-small`). Recommended model for better accuracy You can improve the accuracy of results when converting plain text queries to SQL queries if you use the llama-3-3-70b-instruct model (ID: `llama-3-3-70b-instruct`). However, this model requires significantly more resources than the granite-4-h-small model. For more information about the resources required for each model, see GPU requirements for models. Using other models If you chose to use a different model, the accuracy of the results might vary. Prerequisite This option applies only to environments with local GPUs (`enableModelsOn: gpu`). If you want to use a custom model on a remote instance of watsonx.ai (`enableModelsOn: remote`), see Enabling users to connect to an external IBM watsonx.ai foundation model in the Data Fabric documentation. Default value `granite-4-h-small` Valid values Specify the ID of the model that you want to use. The IDs of the recommended models are: `granite-4-h-small` `llama-3-3-70b-instruct`

Reverting temporary patches

If you applied any patches to your current installation of watsonx.data intelligence, check the patch instructions for cleanup steps and complete these before you start the upgrade. The patch instructions will contain cleanup instructions similar to the ones in this example: Installing the patch for version 5.0.3 in the IBM Cloud Pak® for Data 5.0 documentation.

Upgrading the service

To upgrade watsonx.data intelligence:

Log the cpd-cli in to the Red Hat OpenShift Container Platform cluster:
```
${CPDM_OC_LOGIN}
```
Remember: CPDM_OC_LOGIN is an alias for the cpd-cli manage login-to-ocp command.

Update the operator and custom resource for watsonx.data intelligence.

Run the appropriate command to create the custom resource.

Default installation (without installation options)

cpd-cli manage install-components \
--license_acceptance=true \
--components=watsonx_dataintelligence \
--release=${VERSION} \
--patch_id=${PATCH_ID} \
--operator_ns=${PROJECT_CPD_INST_OPERATORS} \
--instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--image_pull_prefix=${IMAGE_PULL_PREFIX} \
--image_pull_secret=${IMAGE_PULL_SECRET} \
--upgrade=true

Custom installation (with installation options)

cpd-cli manage install-components \
--license_acceptance=true \
--components=watsonx_dataintelligence \
--release=${VERSION} \
--patch_id=${PATCH_ID} \
--operator_ns=${PROJECT_CPD_INST_OPERATORS} \
--instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--image_pull_prefix=${IMAGE_PULL_PREFIX} \
--image_pull_secret=${IMAGE_PULL_SECRET} \
--param-file=/tmp/work/install-options.yml \
--upgrade=true

Validating the upgrade

watsonx.data intelligence is upgraded when the install-components command returns:

[SUCCESS]... The install-components command ran successfully

If you want to confirm that the custom resource status is Completed, you can run the cpd-cli manage get-cr-status command:

cpd-cli manage get-cr-status \
--cpd_instance_ns=${PROJECT_CPD_INST_OPERANDS} \
--components=watsonx_dataintelligence

What to do next

You must complete the following tasks before users can access the service:

The Analytics Engine powered by Apache Spark service is also upgraded automatically, but the instance for Analytics Engine powered by Apache Spark must be upgraded manually.
Complete the steps that apply to the model deployment mode for your installation:
enableModelsOn: cpu
If the inference foundation models component (watsonx_ai_ifm) is not used by any other service or as a remote instance (see Connecting to a remote watsonx.ai instance), you can delete the watsonx_ai_ifm custom resource:
```
oc delete watsonxaiifm watsonxaiifm-cr -n "${PROJECT_CPD_INST_OPERANDS}"
```
After successful deletion, you can see the following output:
```
watsonxaiifm.watsonxaiifm.cpd.ibm.com "watsonxaiifm-cr" deleted
```
If the inference foundation models component (watsonx_ai_ifm) is used by another service or as a remote instance, manually remove the ibm/granite-3-8b-instruct and granite-4-h-small models as described in Removing foundation models from IBM watsonx.ai.
enableModelsOn: gpu

After the upgrade, granite-4-h-small is set as the new default model for the deployment mode gpu. Unless you have sufficient GPUs to host the previously used and the new model, remove the ibm/granite-3-8b-instruct model as described in Removing foundation models from IBM watsonx.ai.

enableModelsOn: remote
If the inference foundation models component (watsonx_ai_ifm) is not used by any other service or as a remote instance (see Connecting to a remote watsonx.ai instance), you can delete the watsonx_ai_ifm custom resource:
```
oc delete watsonxaiifm watsonxaiifm-cr -n "${PROJECT_CPD_INST_OPERANDS}"
```
After successful deletion, you can see the following output:
```
watsonxaiifm.watsonxaiifm.cpd.ibm.com "watsonxaiifm-cr" deleted
```
In this version, the foundation model that is used to generate semantic embeddings was changed to a multilingual embedding model. If semantic embeddings were enabled in the previous version (enableSemanticEmbedding: true), you must regenerate all embeddings for continued use of the Text-to-SQL capabilities. For more information, see Updating projects and metadata with a new embedding model after upgrading.