Choosing compute resources for running tools in projects

You use compute resources in projects when you run jobs and most tools. Depending on the tool, you might have a choice of compute resources for the runtime for the tool.

Compute resources are known as either environment templates or hardware and software specifications. In general, compute resources with larger hardware configurations incur larger usage costs. Many tools in projects use the watsonx.ai Studio service for compute resources, but some tools use other services. Each service tracks and bills compute usage separately.

These tools have multiple choices for configuring runtimes that you can choose from:

These tools have one runtime configuration that is assigned automatically:

The following tools do not consume compute resources:

  • Master Data configuration

For details about how much CUH is consumed for various tools and processes and CUH usage pricing, see Billing details for for watsonx.ai Studio tools and Billing details for machine learning assets.

Metadata import

The amount of CUH per hour that metadata import jobs consume depends on the number of assets that are imported. For each 50,000 assets, 1 CUH is consumed.

IBM Knowledge Catalog legacy plans

Metadata import does not consume compute resources.

Metadata enrichment

The amount of CUH per hour that metadata enrichment jobs consume depends on the enrichment objectives that you select.

CUH usage in specific metadata enrichment objectives
Metadata enrichment objectives Capacity units per hour (CUH)
Profile data 4
Expand metadata 1.5 per 1,000 asset with 100 columns each
Assign terms and classifications 0.75 per 1,000 assets
Use of gen AI for term assignment will incur additional charges.
Set relationships 4
Identify data quality checks 0.06 per 1,000 assets with 100 columns each
Run data quality analysis 4

You can track compute usage for the metadata enrichment objectives on the Resource usage page of the Manage tab of the project.

CUH consumption for profiling and running data quality analysis is aggregated and shown under Profiling. CUH consumption for other selected objectives in metadata enrichment is aggregated and shown under Metadata enrichment.

IBM Knowledge Catalog legacy plans

The amount of CUH per hour from IBM watsonx.data intelligence that metadata enrichment jobs consume depends on the enrichment objectives that you select.

Table 1. CUH usage in specific metadata enrichment objectives
Metadata enrichment objectives Capacity units per hour (CUH)
Profile data 6
Profile data and assign terms 8

When you run metadata enrichment, one or more jobs are started. Each job handles a maximum of 200 tables. When you enrich more than 200 tables at a time, you start multiple jobs. For example, if you run metadata enrichment on 500 tables, you start three jobs. The minimum amount of CUH that is billed for each metadata enrichment job is 0.96 CUH.

Jobs for metadata enrichment with the Expand metadata option or semantic term assignment are limited to 10 tables per job.

The amount of CUH consumed by metadata enrichment depends on the number of tables, as well as columns in the tables. Other factors, such as the structure of the data, can also affect the amount of consumed CUH. For example:

  • The three jobs for profiling data for 500 tables with 500 columns might consume a total of approximately 24 CUH.
  • The three jobs for profiling data and assigning terms for 500 tables with 500 columns might consume a total of approximately 30 CUH.

The runtimes for metadata enrichment do not appear on the Resource usage page of the Manage tab of the project. You can't track compute usage for metadata enrichment.

Advanced profiling

This type of analysis requires the DataStage service. Advanced profiling runs as a DataStage flow with the Default DataStage PX S environment, which consumes 1 CUH per hour, with a minimum of 1 minute of CUH.

The runtime for advanced profiling appears as a DataStage flow on the Resource usage page of the Manage tab of the project.

Primary key or in-depth relationship analysis

These types of analysis require the DataStage service. A primary key or in-depth relationship analysis runs as a DataStage flow with the Default DataStage PX S environment, which consumes 1 CUH per hour, with a minimum of 1 minute of CUH.

The runtime for primary key or in-depth relationship analyses appears as a DataStage flow on the Resource usage page of the Manage tab of the project.

Data quality rules

Data quality rules require the DataStage service. A data quality rule job runs as a DataStage flow with the Default DataStage PX S environment, which consumes 1 CUH per hour, with a minimum of 1 minute of CUH.

The runtime for data quality rules appears as a DataStage flow on the Resource usage page of the Manage tab of the project.

Profiling data assets

Profiling a data asset in a project or a catalog consumes 4 CUH per hour.

You can track compute usage for profiling outside of metadata enrichment on the Resource usage page of the Manage tab of the project.

IBM Knowledge Catalog legacy plans

Profiling a data asset in a project or a catalog consumes 6 CUH per hour from the IBM watsonx.data intelligence, with a minimum amount of 0.96 CUH per profiling session.

The runtime for profiling does not appear on the Resource usage page of the Manage tab of the project. You can't track compute usage for profiling.

Changing the environment runtime for a tool

For tools that support multiple runtime environments, you can select a larger environment runtime if you notice that processing is slow.

To change an environment runtime:

  1. Save any data from your current session before switching to another environment.
  2. Stop the active runtime under Tool runtimes on the Environments page on the Manage tab of your project.
  3. Restart the tool and select another environment with the compute power and memory capacity that better meets your requirements.

Learn more