What's new in IBM Cloud Pak for Data?

Refresh 9 of Version 4.0 platform includes various fixes.

For details, see What's new and changed in the platform.

Version 4.0.9 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

Version 1.3.5 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

The 4.0.9 release of Analytics Engine Powered by Apache Spark includes the following features and updates:

Spark 3.2 is now added, and Spark 3.0 is deprecated

You can start using Spark 3.2 to run your applications on Analytics Engine Powered by Apache Spark instances. Although you can still use Spark 3.0, you should think about moving to Spark 3.2.

Python 3.8 has been removed

Python 3.8 has been removed on IBM Z® and LinuxONE. You can run your Spark applications with Python 3.9.

Version 4.0.9 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

Version 4.0.9 of the Cognos Dashboards service includes various fixes.

For details, see What's new and changed in Cognos Dashboards.

Version 4.0.9 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

Version 4.0.9 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

The 4.0.9 release of DataStage includes the following features and updates:

Connect to more data sources in DataStage

You can now include data from these data sources in your DataStage flows:

• Generic S3
• Teradata (optimized)

For the full list of connectors, see DataStage connectors.

Certain connectors now provide a faster way to test and add metadata from their associated connections

When you create the connection, the Test connection button on Add connection page now works for these connections. (Previously, you did not have a way to test the connection in the user interface.)

• Apache Kafka
• Db2® (optimized)
• Netezza® Performance Server (optimized)
• ODBC
• Oracle (optimized)
• Salesforce.com (optimized)
• Teradata (optimized)

After you create the connection, in DataStage you can drag the Asset browser to the canvas, select a connection and drill down to add or preview the data for these connectors. (Previously, your only option was to drag a connector to the canvas, double-click it to open its Details card, and then go to Properties > Connection and select the connection.)

• Db2 (optimized)
• ODBC

For the full list of connectors, see DataStage connectors.

New stages

The following stages are now available for you to use in DataStage flows:

• Complex Flat File (CFF)
• Hierarchical stage: REST step
• Build stage
• Match Frequency stage
• One-source Match stage

For more information and the full list of stages, see DataStage stages and QualityStage stages. For more information on the Build stage, see Defining build stages.

Download a DataStage flow and its dependencies as a single file

You can download an individual DataStage flow and its dependencies conveniently bundled together as a ZIP file. You can then import the file into another project.

Dependencies include items such as connections, subflows, and parameter sets.

For details, see Downloading and importing a DataStage flow and its dependencies.

Version 4.0.9 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

Deprecation notice for do_12.9 and do_12.10 CPLEX runtimes

The CPLEX runtimes do_12.9 and do_12.10 will be removed in the next release of Cloud Pak for Data, so you must start using the do_20.1 runtime.

Important: Make sure that all your existing deployments use do_20.1 before you upgrade to the next release of Cloud Pak for Data. Any outdated deployments will be removed during the upgrade to the next release.

You can update your existing deployments by using either of the following options:

• REST API
• UI

Version 4.0.9 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

Version 4.0.9 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

Version 1.1.2780 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

Open Data for Industries is now available on Cloud Pak for Data Version 4.0. Use the Open Data for Industries service on Cloud Pak for Data to govern your oil and gas data.

New tool to install the software utilities

The installation process for the software utilities is simplified with the new operator-based tool. In addition, the configuration and validation process is now automated. For details, see Installing software utilities.

Support for multitenancy resource separation

With resource separation, you can reduce operational expenses while maintaining security, compliance, and independent operability. For details, see Multitenancy in Open Data for Industries.

Integration of registration and notification

Use the newly integrated notification mechanism in Open Data for Industries to enable the API endpoints to communicate event-driven messages to internal and external data solutions. For details, see Notifications in Open Data for Industries.

Documentation enhancements

The Getting started with Open Data for Industries documentation guides you through the pre-installation, installation, configuration, and usage tasks.

Version 4.0.9 of the Planning Analytics service includes various fixes.

Version 1.0.4 of the Product Master service includes various fixes.

For details, see What's new and changed in Product Master.

Version 4.0.9 of the RStudio Server with R 3.6 service includes various fixes.

For details, see What's new and changed in RStudio Server with R 3.6.

Version 4.0.9 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

The 4.0.9 release of Watson Discovery includes the following features and updates:

API usage information is now available from the user interface

You can now get information about the analyze API usage from the API usage page (Data usage > API usage) in the Watson Discovery user interface. For details about the API, see Analyze API.

A new document status API is supported in Cloud Pak for Data instances

Use the new document status API to programmatically get a list of the documents in a collection and to get details about a single document.

The API is supported for collections that are created after 23 March 2022. If you want to get status information about a collection that was created earlier, you can trigger a process that runs the conversion step of ingestion on the documents. For example, from the Activity page for the collection, click Recrawl.

The API is not supported from the SDKs.

For more information about the new API, see the API reference documentation.

Version 4.0.9 of the Watson Discovery service includes various fixes.

Version 4.0.9 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

Version 4.0.9 of the Watson Knowledge Studio service includes various fixes.

Version 4.0.9 of the Watson Machine Learning service includes various fixes.

Version 2.3.9 of the Watson Machine Learning Accelerator service includes various fixes.

Version 4.0.9 of the Watson OpenScale service includes various fixes.

For details, see What's new and changed in Watson OpenScale.

Version 4.0.9 of the Watson Speech to Text service includes various fixes.

Version 4.0.9 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

Version 4.0.9 of the Watson Studio Runtimes service includes various fixes.

For details, see What's new and changed in Watson Studio Runtimes.

Version 4.0.9 of the Watson Text to Speech service includes various fixes.

New best practice for installing Cloud Pak for Data

The Cloud Pak for Data documentation includes a new best practices topic on setting up a script to source environment variables with the appropriate information for your environment. The script enables you to run most of the commands in the Installing and Upgrading sections of the documentation without making any changes.

The script can also help you:

• Improve the consistency of installations across multiple environments
• Simplify the process of managing multiple deployments on the same cluster

Restriction: The code blocks in the individual service installation and upgrade topics do not currently use the environment variables, but you can replace variables in the code blocks with the appropriate environment variable before you run the commands.

For details, see Best practices: Setting up install variables.

Updated guidance on pinning the control plane at a specific version

The documentation for installing and upgrading the Cloud Pak for Data control plane includes updated guidance for pinning the control plane at a specific version. This guidance helps prevent unexpected upgrades and is especially important if either of the following situations applies to your installation:

• You have multiple IBM Cloud Paks on your cluster
• You have multiple instances of IBM Cloud Pak for Data on your cluster

In these scenarios, you might need to support different versions of the ZenService control plane for each installation.

Refresh 8 of Version 4.0 platform includes various fixes.

Version 4.0.8 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

Version 1.3.4 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

The 4.0.8 release of Analytics Engine Powered by Apache Spark includes the following features and updates:

Support for Spark service CLI commands

You can now use Analytics Engine Powered by Apache Spark CLI commands for:

• Instance management actions
• Spark application actions
• History server actions

For details, see the Spark service CLI.

Python 3.7 and 3.8 have been removed

Python 3.7 and Python 3.8 have been removed in this release. You can no longer run applications by using Python 3.7 or 3.8. You must use Python 3.9.

Python 3.9 added to IBM Z and LinuxONE

You can now run Spark applications with Python 3.9 on IBM Z and LinuxONE

Support for Power® hardware has been removed

You can no longer install Analytics Engine Powered by Apache Spark on Red Hat OpenShift Container Platform clusters running on Power hardware.

Version 4.0.8 of the Analytics Engine Powered by Apache Spark includes various fixes.

Version 4.0.8 of the Cognos Analytics service includes various fixes.

The 4.0.8 release of Cognos Dashboards includes the following features and updates:

Select multiple objects by dragging your mouse

You can now select multiple widgets and visualizations by using your mouse. Click anywhere on the canvas and drag your cursor around the objects that you want to select. All items within the lasso are selected, and you can group them or easily move them all.

Vertical and horizontal distribution for objects

You can specify a vertical or horizontal distribution for objects on your dashboard to determine where they appear.

Set vertical or horizontal alignment by selecting at least three objects, and then either click the Align icon from the toolbar or use the Visualization properties under Layout.

Change the style on axis and legend text

You now have more options for changing the text styles in visualizations on your dashboard.

Right-click an axis or a legend title and select Style text from the list. You can edit the title and the title style. And, you can apply your styling to all the axes on the visualization.

A different background color for each tab

You now have more control over the look of tabbed dashboards. In the dashboard properties, you can set a different background color for each of your tabs.

Enhanced widget properties

You can now style any visualization or widget on your dashboard to have rounded corners. The opacity property was improved so that it affects only the text widget background and not the text.

In addition, you can now control the following text widget properties:

• You can adjust the inner margins.
• You can choose a vertical alignment.
• You can set indentations.
• You can specify the line height.

Version 4.0.8 of the Cognos Dashboards service includes various fixes.

Version 4.0.8 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

Version 4.0.8 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

Version 1.7.8 of the Data Virtualization service includes various fixes.

For details, see What's new and changed in Data Virtualization.

The 4.0.8 release of DataStage includes the following features and updates:

New connector: SAP HANA

You can now include data from SAP HANA in your DataStage flows. For the full list of DataStage connectors, see DataStage connectors.

Version 4.0.8 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

The 4.0.10 release of Db2 includes the following features and updates:

SSL certificate extended

The SSL endpoint certificate that is used by Db2 now has an expiration of one year and 30 days. Previously the certificate expired after one month. If you are installing Db2 for the first time, the new certificate is automatically installed when you deploy the Db2 service. This feature is only available for new deployments.

Version 4.0.10 of the Db2 service includes various fixes.

Version 7.2.8 of the Db2 Big SQL service includes various fixes.

Version 2.0.8 of the Db2 Data Gate service includes various fixes.

For details, see What's new and changed in Db2 Data Gate.

The 4.0.8 release of Db2 Data Management Console includes the following features and updates:

TLS certificate support for Db2 SSL

Db2 Data Management Console now supports Db2 and Db2 Warehouse service instances that are configured to use the new TLS certificate for SSL connections.

The new TLS certificate has an expiration of one year and 30 days. Previously the certificate expired after one month.

Version 4.0.8 of the Db2 Data Management Console service includes various fixes.

For details, see What's new and changed in Db2 Data Management Console.

The 4.0.10 release of Db2 Warehouse includes the following features and updates:

SSL certificate extended

The SSL endpoint certificate that is used by Db2 Warehouse now has an expiration of one year and 30 days. Previously the certificate expired after one month. If you are installing Db2 Warehouse for the first time, the new certificate is automatically installed when you deploy the Db2 Warehouse service. This feature is only available for new deployments.

Replication is now available on Db2 Warehouse MPP

The replication feature is now enabled for Db2 Warehouse massively parallel processing (MPP) environments. Replication supports continuous availability, data locality, and compliance and regulatory requirements. For details, see Replication.

Version 4.0.10 of the Db2 Warehouse service includes various fixes.

The 4.0.8 release of Decision Optimization includes the following features and updates:

The default Python for Decision Optimization users is now 3.9. Python 3.7 and 3.8 have been removed.

Python is used to run Decision Optimization models formulated in DOcplex (a native Python API for Decision Optimization) in both Decision Optimization experiments and Jupyter notebooks. Modeling Assistant models also use Python because DOcplex code is generated when models are deployed.

Models that are formulated in OPL (a modeling language), or in specific file formats for CPLEX or CP Optimizer (the solver engines), such as LP or CPO formats, are not affected by this Python update.

If your models use the default Python version no change is required because the upgrade will be handled automatically. If however you have explicitly specified Python 3.7 or 3.8 in your model, you must update this version specification.

• For experiments, you can select the Python version in the Run configuration pane. (For details, see Run parameters.) If the Python version parameter is not specified in your experiment, the default version is used automatically, and no action is required.
• For Jupyter notebooks that use older versions of Python, you can change the Python version by selecting Change environment in the list of actions. Before you can edit your notebook, you must select a supported environment. When your notebook is open for editing, you can select an environment definition on the Environment tab in the Information pane in your notebook. For details, see Decision Optimization notebooks.
• If you have an existing deployed model that explicitly uses Python 3.7 or 3.8 you must take one of the following actions:
  • Edit and redeploy your model. For details, see REST API example.
  • Change or remove the Python version by using the REST API without having to redeploy the model. For details, see Changing the Python version for an existing deployed model using the REST API.

Version 4.0.8 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

Version 12.10 and 13.6 of the EDB Postgres service includes various fixes.

Version 4.0.8 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

The Guardium External S-TAP service is now available to monitor compliance and secure the data in your databases. Guardium External S-TAP is a component of IBM Guardium Data Protection.

With this release, you can install and configure the Guardium External S-TAP service in high-availability mode to intercept TCP/IP traffic (plain-text or encrypted) between Cloud Pak for Data users and database services. The intercepted traffic is sent to the Guardium collector for parsing, policy enforcement, logging, and reporting.

The 1.1.259 release of IBM Match 360 includes the following features and updates:

New data auditing attributes provide data traceability

Now when you add or edit record data in IBM Match 360, the service adds system-generated audit attributes to each record so you can track their audit history, including:

• The date when the data was added to IBM Match 360
• The date when the data was last updated
• The user who last updated the data

Improve your matching results by using multi-dimensional comparison filters

You can now fine tune your matching algorithm by defining multi-dimensional comparison filters. You can use these filters to compare selected attributes and then adjust matching scores and weights up or down based on criteria that you define. Multi-dimensional comparison filters can reduce the amount of false positive or false negative matches in your matching results. For details, see Customizing and strengthening your matching algorithm.

Version 1.1.2590 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

Version 4.1.1 of the Informix service includes various fixes.

Versions 4.2.6 and 4.4.0 of the MongoDB service include various fixes.

Version 8.204.5 of the OpenPages service includes various fixes.

Version 4.0.8 of the Planning Analytics service includes various fixes.

The 4.0.8 release of RStudio Server with R 3.6 includes the following features and updates:

Support for R 3.6 on IBM Runtimes 22.1

You can now use an IBM Runtimes 22.1 on R 3.6 environment when you work on R notebooks and scripts. This R 3.6 environment is based on the IBM Runtimes 22.1 release. For information on the IBM Runtimes 22.1 release, see Environments.

Version 4.0.8 of the RStudio Server with R 3.6 service includes various fixes.

For details, see What's new and changed in RStudio Server with R 3.6.

Version 4.0.8 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

Version 1.0.8 of the Voice Gateway service includes security fixes and the following update:

Removed support for TLS v1.1

Voice Gateway no longer supports Transport Layer Security (TLS) v1.1.

Version 4.0.8 of the Watson Assistant service includes security fixes and the following updates:

Closed entity values in French are normalized so that values match regardless of accent marks

Closed entities exact matches in French now use accent-normalized values or synonyms. For example, if you define a closed entity with a value or synonym with accent marks (for example, garçon or déjà), then variants without accent marks are also recognized (garcon or deja). Likewise, if a closed entity value or synonym is defined without accent marks, then user inputs with accent marks are also recognized.

Pattern entities allow spelling autocorrection

Pattern entities that match all characters and words that are usually used to count input words allow spelling autocorrection. For example, if a user defines the ^..{0,19}$ pattern entity that matches the first 20 characters of an input, then the entity match does not affect spelling autocorrection. In this example, an input of cancl transaction is autocorrected to cancel transaction. This change applies to the following languages: English and French.

Fuzzy matching updates

Previously, an update was made so that interactions between the stemming and misspelling fuzzy matching features were not allowed in English, French, German, and Czech. These interactions are still not allowed in English, but are allowed in French, German, and Czech.

Improved irrelevance detection for Dutch

Irrelevance detection for Dutch disregards any punctuation in an input sentence. For example, you can now expect the same confidence score for the following two inputs: ik ben een kleine krijger? and ik ben een kleine krijger. In this example, the question mark (?) doesn't affect the confidence score.

Improved enhanced intent detection

The exact match in enhanced intent detection now better handles small differences between training examples and runtime utterances when the differences do not change the meaning of a sentence.

The 4.0.8 release of Watson Discovery includes the following features and updates:

The Development deployment type was changed to Starter

When you install Watson Discovery, you can optionally specify the type of deployment by including the deploymentType parameter in your custom resource. The Development option is now called the Starter option.

The Development and Starter options are functionally the same, and both values are accepted by the service.

Version 4.0.8 of the Watson Discovery service includes various fixes.

The 4.0.8 release of Watson Knowledge Catalog includes the following features and updates:

Filtering workflow tasks

In your Task inbox, you can now filter your open, in progress, and completed tasks so that you can find the tasks you want faster. Several filters are available, including request type and task status.

Version 4.0.8 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

The 4.0.8 release of Watson Machine Learning includes the following features and updates:

Python 3.7 and 3.8 software specifications discontinued

Support for frameworks and software specifications based on Python 3.7 and 3.8 are discontinued in this release. Use software specifications based on Python 3.9 instead. For details, see Deployment frameworks and software specifications.

Train AutoAI experiment with data from a Generic S3 connection

You can now train an AutoAI experiment with data from a Generic S3 connection. This connection is available for training only and is not yet available for input data when you deploy the resulting model.

Support is discontinued for Core ML deployments for iOS

Core ML deployments for iOS, also known as virtual deployments, are no longer a supported deployment type.

Version 4.0.8 of the Watson Machine Learning service includes various fixes.

The 2.3.8 release of Watson Machine Learning Accelerator includes the following features and updates:

Use the latest deep learning frameworks

Watson Machine Learning Accelerator now supports the following deep learning libraries:

• TensorFlow 2.7.1
• PyTorch 1.10
• torchVision 0.11.3
• OpenCV 4.5.5
• xgboost 1.5.2

Access the Watson Machine Learning Accelerator service console from the Instances page

The URL to the Watson Machine Learning Accelerator console is now available on the Instances page. Use the URL to quickly access the service instance console.

For details on accessing the console, see Deep learning with Watson Machine Learning Accelerator.

Version 2.3.8 of the Watson Machine Learning Accelerator service includes various fixes.

Version 4.0.8 of the Watson OpenScale service includes various fixes.

For details, see What's new and changed in Watson OpenScale.

Version 4.0.8 of the Watson Speech to Text service includes various fixes.

The 4.0.8 release of Watson Studio includes the following features and updates:

Python 3.7 and Python 3.8 have been removed

Python 3.7 and Python 3.8 have been removed in Refresh 8 of Version 4.0. If you want to continue using your Python 3.7 and Python 3.8 notebooks, see Migrating Python 3.7 and 3.8 environments to Refresh 8 of Cloud Pak for Data 4.0 before you upgrade to Refresh 8 of Version 4.0.

Support for R 3.6 on IBM Runtimes 22.1

You can now use an IBM Runtimes 22.1 on R 3.6 environment when you work on R notebooks and scripts. This R 3.6 environment is based on the IBM Runtimes 22.1 release. For information on the IBM Runtimes 22.1 release, see Environments.

Deprecation notice for Default R 3.6

The Default R 3.6 environment is deprecated and will be removed in an upcoming release. Start using the IBM Runtimes 22.1 on R 3.6 environment when you create R notebooks and scripts.

JupyterLab 3.3.1 with Python Debugger enabled

JupyterLab is upgraded to Version 3.3.1 in Python runtimes and the debugger for ipykernel for Jupyter is enabled. You can now debug your Python notebooks and python script files directly from within JupyterLab.

Version 4.0.8 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

The 4.0.8 release of Watson Studio Runtimes includes the following features and updates:

Python 3.7 and Python 3.8 have been removed

Python 3.7 and Python 3.8 have been removed in Refresh 8 of Version 4.0. If you want to continue using your Python 3.7 and Python 3.8 notebooks, see Migrating Python 3.7 and 3.8 environments to Refresh 8 of Cloud Pak for Data 4.0 before you upgrade to Refresh 8 of Version 4.0.

Support for R 3.6 on IBM Runtimes 22.1

You can now use an IBM Runtimes 22.1 on R 3.6 environment when you work on R notebooks and scripts. This R 3.6 environment is based on the IBM Runtimes 22.1 release. For information on the IBM Runtimes 22.1 release, see Environments.

Deprecation notice for Default R 3.6

The Default R 3.6 environment is deprecated and will be removed in an upcoming release. Start using the IBM Runtimes 22.1 on R 3.6 environment when you create R notebooks and scripts.

JupyterLab 3.3.1 with Python Debugger enabled

JupyterLab is upgraded to Version 3.3.1 in Python runtimes and the debugger for ipykernel for Jupyter is enabled. You can now debug your Python notebooks and python script files directly from within JupyterLab.

Version 4.0.8 of the Watson Studio Runtimes service includes various fixes.

For details, see What's new and changed in Watson Studio Runtimes.

Version 4.0.8 of the Watson Text to Speech service includes various fixes.

Refresh 7 of Version 4.0 platform includes various fixes.

For details, see What's new and changed in the platform.

Version 4.0.7 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

Version 1.3.3 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

The 4.0.7 release of Analytics Engine Powered by Apache Spark includes the following features and updates:

Spark 2.4 has been removed

Spark 2.4 has been removed in this release. You can no longer run Spark applications by using Spark 2.4. You must use Spark 3.0.

Deprecation notice for Python 3.7 and 3.8

Python 3.7 and 3.8 is deprecated and will be removed in an upcoming refresh. Start using Python 3.9 when creating Spark applications.

Version 4.0.7 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

Version 4.0.7 of the Cognos Analytics service includes various fixes.

Version 4.0.7 of the Cognos Dashboards service includes various fixes.

Version 4.0.7 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

The 4.0.7 release of Data Refinery includes the following features and updates:

Data Refinery flow jobs that use an environment definition with Spark 2.4 must be updated

If you have a Data Refinery flow job that uses Spark 2.4, for example, the "Default Spark 2.4 & R 3.6" environment definition, the job will fail. Change the environment definition to "Default Spark 3.0 & R 3.6," "Default Data Refinery XS," or create your own "Spark 3.0 & R 3.6" environment definition. For information, see Data Refinery environments.

Version 4.0.7 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

Use filters to make loading and virtualizing of tables faster

On the Data sources page, you can now filter tables for each data source based on catalog, schema, or table name with the new Filter data action in the overflow menu. Filters take effect immediately. When you virtualize data, only assets that meet the filter criteria are displayed, which speeds up loading and virtualization of tables.

For more information, see Filtering data in Data Virtualization.

When you virtualize objects, Data Virtualization loads a limited number of data sources to improve performance. You can filter the data sources to load by Source type, Database, and Schema to quickly find only those data sources that you are interested in.

You can click to refresh your data sources or load newly added data sources. Notifications indicate when new data sources become available and when tables are updated.

For more information, see Creating virtual objects in Data Virtualization.

Enhanced statistics collection

You can gather statistics on a virtualized table by using the new DVSYS.COLLECT_STATISTICS procedure. This procedure replaces the SYSPROC.NNSTAT procedure and incorporates the following improvements:

• Collect statistics from remote data sources that support statistics collection.
• The number of null values in a column of a table.

For more information, see Collecting statistics in Data Virtualization.

Support for column headers in flat files in Cloud Object Storage

You can now virtualize text files that contain column headers in data sources in Cloud Object Storage. Column headers help categorize the data in columns for readability. For more information, see Creating a virtualized table from files in Cloud Object Storage in Data Virtualization.

Support for credentials that are stored in a vault for remote connections

You can now use credentials that are stored in a vault with remote connectors in Data Virtualization. For more information, see Using secrets from vaults in connections and Accessing data sources by using remote connectors in Data Virtualization.

Support for more data sources

• You can connect to TM1 databases that store data in multidimensional OLAP cubes by using the Planning Analytics connection type.
• You can use the SAP OData connection type to extract data from an SAP Business Warehouse (BW) or an SAP S/4HANA data source through its exposed OData services.

For more information, see Supported data sources in Data Virtualization.

Support for UNIQUE and DISTINCT predicate pushdown on more data sources

Predicate pushdown is an optimization that reduces query times and memory usage. The following data sources now support pushdown of UNIQUE and DISTINCT predicates: Amazon Redshift, Apache Derby, Cloudera Impala, Data Virtualization Manager, Db2, Db2 Big SQL, Denodo, Google BigQuery, Greenplum, Hive, Informix, MariaDB, Microsoft SQL Server, MongoDB, Netezza, Oracle, PostgreSQL, Salesforce.com, SAP ASE, SAP HANA, SAP OData, Snowflake, SparkSQL using a Generic JDBC connection, Teradata.

Version 1.7.7 of the Data Virtualization service includes various fixes.

For details, see What's new and changed in Data Virtualization.

The 4.0.7 release of DataStage includes the following features and updates:

New connectors

You can now include data from these data sources in your DataStage flows:

• IBM MQ
• Microsoft Azure Cosmos DB
• Microsoft Azure SQL Database

For the full list of DataStage connectors, see DataStage connectors.

Version 4.0.7 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

Version 4.0.9 of the Db2 service includes various fixes.

Version 7.2.7 of the Db2 Big SQL service includes various fixes.

The 2.0.7 release of Db2 Data Gate includes the following features and updates:

New synchronization details available

You can now view the number of inserted rows, updated rows, deleted rows, rollback rows, and utility rows on the Db2 Data Gate dashboard. You can use this information to evaluate the synchronization workload and analyze performance bottlenecks. For details, see Monitoring a Db2 Data Gate instance.

New button to temporarily disable synchronization

You can now click a button on the dashboard to disable synchronization temporarily for the entire set of tables in a Db2 Data Gate instance. Temporarily disabling synchronization is useful during maintenance jobs because pending updates do not get lost in the process, but can be applied when synchronization is re-enabled. For details, see Monitoring a Db2 Data Gate instance.

Alias for Db2 Data Gate instance name

Each time you create a Db2 Data Gate instance, you can specify a Data Gate pairing name, which is an alias for the instance name. The pairing name makes it easier to specify the correct instance name in routed queries and stored procedure calls because the original instance names can be long and cryptic. For details, see Creating a Db2 Data Gate instance.

Version 2.0.7 of the Db2 Data Gate service includes various fixes.

Version 4.0.7 of the Db2 Data Management Console service includes various fixes.

For details, see What's new and changed in Db2 Data Management Console.

The Db2 Event Store service deprecated

The Db2 Event Store service is deprecated and will be removed from a future release of Cloud Pak for Data.

If you want to replace Db2 Event Store functionality, contact your IBM sales representative to discuss your options.

Version 4.0.9 of the Db2 Warehouse service includes various fixes.

The 4.0.7 release of Decision Optimization includes the following features and updates:

Deprecation notice for Python 3.7 and 3.8

Python 3.7 and 3.8 is deprecated and will be removed in an upcoming refresh. Start using Python 3.9.

Version 4.0.7 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

Version 12.10 and 13.6 of the EDB Postgres service includes various fixes.

Version 4.0.7 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

The 1.1.215 release of IBM Match 360 includes the following features and updates:

Gain new insight by matching records on more than one entity type

Each record in your data can now be matched to multiple entity types. Each entity type that you define has its own matching algorithm for you to configure, tune, and run to create new types of entities. You can now run multiple matching algorithms on your record data to gain insight and discover connections within your data.

For example, in addition to resolving person record data into person entities by using a matching algorithm for deduplication, you can also resolve the same person data into meaningful new types of entities such as households or families.

For details, see Matching your data to create master data entities.

Predefined scaling options

Now you can use predefined deployment sizes to scale your IBM Match 360 deployment. Choose the size that meets your needs. For details, see Scaling services.

Version 1.1.2150 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

Versions 4.2.6 and 4.4.0 of the MongoDB service include various fixes.

Version 8.204.4 of the OpenPages service includes various fixes.

Version 4.0.7 of the Planning Analytics service includes various fixes.

Version 4.0.7 of the RStudio Server with R 3.6 service includes various fixes.

For details, see What's new and changed in RStudio Server with R 3.6.

Version 4.0.7 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

Version 1.0.7 of the Voice Gateway service includes various fixes.

Version 4.0.7 of the Watson Assistant service includes security fixes and the following updates:

Enhanced irrelevance detection update

We revised the enhanced irrelevance detection classification algorithm. Now, enhanced irrelevant detection uses any provided counterexamples in training. This change does not affect workspaces without counterexamples. This update applies to the following languages: English, French, Spanish, Italian, and the universal language model.

Fuzzy matching updates

Interactions between the stemming and misspelling fuzzy matching features are not allowed. Improve fuzzy matching behavior by limiting the interactions between different fuzzy matching features. This change applies to the following languages: English, French, German, and Czech.

Version 4.0.7 of the Watson Discovery service includes various fixes.

Version 4.0.7 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

The 4.0.7 release of Watson Knowledge Studio includes the following features and updates:

IBM Cloud Block Storage is now supported

When you install Watson Knowledge Studio, you can specify IBM Cloud Block Storage Gold tier (ibmc-block-gold) as your storage class.

Version 4.0.7 of the Watson Knowledge Studio service includes various fixes.

The 4.0.7 release of Watson Machine Learning includes the following features and updates:

Support for new data sources for AutoAI experiments

Train your AutoAI experiments with data stored in Oracle, Teradata, or Google BigQuery databases.

Better insights into your deployed assets

The Deployments dashboard now provides visual indicators for deprecated assets so you know at a glance when assets need updating.

New way to deploy code packages

You can now browse for code packages from a deployment space and create a deployment job for the package without having to use CPDCTL.

Federated learning now supports Python 3.9

Use a more secure version of Python for your Federated Learning experiments with these frameworks:

• Tensorflow 2.7
• PyTorch 1.10
• Scikit-learn 1.0.2

Python 3.8 and all associated frameworks are being deprecated. Upgrade your Federated Learning experiments to Python 3.9 and implement fully supported frameworks. For more information, see Frameworks and Python version compatibility.

Version 4.0.7 of the Watson Machine Learning service includes various fixes.

The 2.3.7 release of Watson Machine Learning Accelerator includes the following features and updates:

Give additional users administrative access to Watson Machine Learning Accelerator

Previously, only users who had the Administer platform permission on Cloud Pak for Data could administer the Watson Machine Learning Accelerator service. Starting in 2.3.7, you can assign specific platform users the Watson Machine Learning Accelerator Administrator role. For details, see Add users to the Watson Machine Learning Accelerator instance.

Version 2.3.7 of the Watson Machine Learning Accelerator service includes various fixes.

The 4.0.7 release of Watson OpenScale includes the following features and updates:

Upload pre-scored test data

Now you can upload test data that was already scored for a model deployment so that Watson OpenScale does not spend resources scoring the test data a second time. Prediction and probability data are uploaded with the test data. For more information, see Manage model risk.

New fairness metrics and explanation algorithms

You can now use Watson OpenScale Python client version 3.0.14 or later to generate new fairness metrics and additional algorithms to explain model transactions. For more information, see Metrics computation using Python SDK.

Version 4.0.7 of the Watson OpenScale service includes various fixes.

Version 4.0.7 of the Watson Speech to Text service includes various fixes.

The 4.0.7 release of Watson Studio includes the following features and updates:

Deprecation notice for Python 3.8

Python 3.8 is deprecated and will be removed in an upcoming refresh. Start using environments with a newer version of Python, which contains newer versions of popular machine learning libraries.

Spark 2.4 has been removed

Spark 2.4 has been removed in this release. Use Spark 3.0 for any Spark applications that you develop.

Version 4.0.7 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

Version 4.0.7 of the Watson Studio Runtimes service includes various fixes.

Version 4.0.7 of the Watson Text to Speech service includes various fixes.

Refresh 6 of Version 4.0 of the Cloud Pak for Data platform includes various fixes.

Version 4.0.6 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

Version 1.3.2 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

The 4.0.6 release of Analytics Engine Powered by Apache Spark includes the following features and updates:

Python 3.9 is now available

When you run your Spark applications in Analytics Engine Powered by Apache Spark, you can now choose to run them with Python 3.9.

Deprecation notice for Python 3.7

Python 3.7 is deprecated and will be removed in an upcoming release. Use Python 3.9 in the Jupyter notebooks that you develop.

Deprecation notice for Spark 2.4

Spark 2.4 is deprecated and will be removed in an upcoming release. Use Spark 3.0 for any Spark applications that you develop.

Version 4.0.6 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

The 4.0.6 release of Cognos Analytics includes the following features and updates:

Updated software version for Cognos Analytics

This release provides software version 11.1.7 Fix Pak 4 IF 8 of Cognos Analytics.

Version 4.0.6 of the Cognos Analytics service includes various fixes.

The 4.0.6 release of Cognos Dashboards includes the following features and updates:

Lots of new features for your dashboards

Cognos Dashboards includes features such as:

• New dashboard customizations are now available including new borders, grids, labels, and legends options
• You can now use new visualization capabilities, such as donut visualizations and 3D hexbin maps
• You can now export your data to CSV files

To see all the new features in Cognos Dashboards, see What's new and changed in Cognos Dashboards.

Version 4.0.6 of the Cognos Dashboards service includes various fixes.

For details, see What's new and changed in Cognos Dashboards.

Version 4.0.6 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

The 4.0.6 release of Data Refinery includes the following features and updates:

Data Refinery supports refining SAS files with the sas7bdat extension

You can now refine SAS data assets that use the .sas7bdat extension. SAS files are supported as source files only. You cannot use SAS files as a target of a Data Refinery flow.

For the full list of file types that are supported by Data Refinery, see Refining data.

Performance enhancements for GUI operations in flows that use large data assets

The following GUI operations are enhanced to work faster on large data assets:

• Convert column type to Integer when you specify a thousands grouping symbol (comma, dot, or custom)
• Convert column type to Decimal with a comma decimal marker or when you specify a thousands grouping symbol (comma, dot, or custom)
• Text > Trim quotes

Important: If you have existing Data Refinery flows that use these GUI operations, you must update the flows. To update a flow, open it, save it, and run a job for the flow. For details, see Managing Data Refinery flows.

For information about the operations, see GUI operations in Data Refinery.

Version 4.0.6 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

Updated guidance for OpenShift Container Storage users

Starting with Data Virtualization 1.7.6, the recommended storage class for OpenShift Container Storage is ocs-storagecluster-ceph-rbd.

• If you are installing Data Virtualization 1.7.6 or upgrading to Data Virtualization 1.7.6 from IBM Cloud Pak for Data version 3.5, specify the ocs-storagecluster-ceph-rbd storage class during installation or upgrade.
• If you are upgrading to Data Virtualization 1.7.6 from a previous refresh of Cloud Pak for Data 4.0, continue to use ocs-storagecluster-cephfs. You cannot change the storage that is used in an existing deployment.

Version 1.7.6 of the Data Virtualization service includes various fixes.

For details, see What's new and changed in Data Virtualization.

The 4.0.6 release of DataStage includes the following features and updates:

New connectors and stages

You can now include data from these data sources in your DataStage flows:

• Amazon RDS for Oracle
• Box
• Compose for MySQL

For more information, see DataStage connectors.

The following stages are now available for you to use in DataStage flows:

• Combine Records
• Make Subrecords
• Make Vector
• Promote Subrecords
• Split Subrecord
• Split Vector

For more information, see DataStage stages.

Version 4.0.6 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

The 4.0.8 release of Db2 includes the following features and updates:

Save space by compressing your backup images

When you install Db2, you can specify that the service enable compression of backup images. Backup compression can help reduce the storage cost in your database environment. If you want to compress backup images, you must specify enableBackupCompression: Yes in the Db2 custom resource.

Version 4.0.8 of the Db2 service includes various fixes.

The 7.2.6 release of Db2 Big SQL includes the following features and updates:

Connect to more Hadoop clusters

You can now connect Db2 Big SQL to Hadoop clusters that are secured with Kerberos even when the Kerberos configuration is not managed by the Cloudera cluster manager server. For details, see Connecting Db2 Big SQL to a Hadoop cluster with a manually managed Kerberos configuration.

Updated guidance for OpenShift Container Storage users

Starting with Db2 Big SQL 7.2.6, the recommended storage class for OpenShift Container Storage is ocs-storagecluster-ceph-rbd.

• If you are installing Db2 Big SQL 7.2.6 or upgrading to Db2 Big SQL 7.2.6 from IBM Cloud Pak for Data version 3.5, specify the ocs-storagecluster-ceph-rbd storage class during installation or upgrade.
• If you are upgrading to Db2 Big SQL 7.2.6 from a previous refresh of Cloud Pak for Data 4.0, continue to use ocs-storagecluster-cephfs. You cannot change the storage that is used in an existing deployment.

Version 7.2.6 of the Db2 Big SQL service includes various fixes.

Version 2.0.6 of the Db2 Data Gate service includes various fixes.

Version 4.0.6 of the Db2 Data Management Console service includes various fixes.

For details, see What's new and changed in Db2 Data Management Console.

The 4.0.8 release of Db2 Warehouse includes the following features and updates:

Save space by compressing your backup images

When you install Db2 Warehouse, you can specify that the service enable compression of backup images. Backup compression can help reduce the storage cost in your database environment. If you want to compress backup images, you must specify enableBackupCompression: Yes in the Db2 Warehouse custom resource.

Version 4.0.8 of the Db2 Warehouse service includes various fixes.

The 4.0.6 release of Decision Optimization includes the following features and updates:

New Python version 3.9

Python 3.9 is now supported for building and deploying Decision Optimization models. The default version remains 3.8. For details, see Decision Optimization notebooks. To use Python 3.9 in your Decision Optimization experiment, select this version in the Run configuration parameters drop-down menu. If you do not select a Python version, the default version 3.8 is used. For details, see Run configuration.

To modify or redeploy any existing deployed Python or Modeling Assistant models to use Python 3.9, use the oaas.docplex.python parameter to specify Python version 3.9.

New Python client library

When you upgrade from Cloud Pak for Data 3.5, the old Python client library dd-scenario is replaced by the decision_optimization_client Python library. You must update any 3.5 notebooks that you still plan to use, if they use the old Python client library.

Version 4.0.6 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

The 12.9 and 13.5 release of EDB Postgres includes the following features and updates:

Custom username and password creation

You can now choose to generate custom user names and passwords for each EDB Postgres instance. Credentials can still be automatically generated, if preferred. For details, see Creating a custom EDB Postgres user name and password.

Version 12.9 and 13.5 of the EDB Postgres service includes various fixes.

For details, see What's new and changed in EDB Postgres.

The 4.0.6 release of Execution Engine for Apache Hadoop includes the following features and updates:

Apache Livy 0.8.0 is now supported

When you use Spark 3.0 with Execution Engine for Apache Hadoop, you can run Apache Livy 0.8.0. When you run Livy 0.8.0, you can create Livy sessions with Python 3.8, which you could not do with Livy 0.7.0.

Version 4.0.6 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

Version 1.1.1880 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

The 4.1.0 release of Informix includes the following features and updates:

Security context enhancement

Informix no longer requires the privileged security context constraint (SCC). Informix can now run with the restricted SCC.

Informix still requires the anyuid SCC when running with the pre-allocated (non-root) userid in the pods.

New ingress network policy

The pods of the Informix instance will now run with an ingress network policy that limits incoming network packets to the ports on which Informix is expecting connections.

Version 4.1.0 of the Informix service includes various fixes.

Version 8.204.3 of the OpenPages service includes various fixes.

Version 4.0.6 of the Planning Analytics service includes various fixes.

Version 4.0.6 of the RStudio Server with R 3.6 service includes various fixes.

For details, see What's new and changed in RStudio Server with R 3.6.

The 4.0.6 release of SPSS Modeler includes the following features and updates:

New scripting documentation

Although scripting or coding isn't required to use SPSS Modeler, scripting can be a powerful tool for automating processes in the user interface. Scripts can perform the same types of actions that you perform with a mouse or a keyboard, and you can use them to automate tasks that would be highly repetitive or time consuming to perform manually.

A new scripting and automation guide describes this feature in detail. You can use this valuable guide to quickly find information such as the following, plus much more:

• Details about all available node properties
• Scripting tips
• Information about the scripting API
• How to use scripting to run a flow
• How to use scripting to reference, create, link and unlink, import, replace, or delete nodes
• Script examples

New data connections are available

You can now create connections for the following data sources and use them in your SPSS Modeler flows. For more information, see Supported data sources for SPSS Modeler.

• Databases for DataStax (import and export supported)
• Exasol (import and export supported; SQL pushback supported with a custom image)
• SQL Query (import supported)

Exasol connections are now supported

When you save a model, a new Software specifications drop-down is now available. If you use Exasol data, you can use this option to select a custom software specification that supports Exasol in SPSS Modeler. Exasol data is supported only in batch deployment jobs at this time.

To populate the new drop-down, see the following information about creating a custom flow session image and a deployment runtime image with the Exasol ODBC driver:

• Building custom images to install ODBC drivers
• Creating and uploading a custom image
• Working with custom images
• Creating a new base software specification for the custom image

Version 4.0.6 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

Version 4.0.6 of the Watson Assistant service includes security fixes and the following updates:

Configure webhook timeout

From the Pre-message webhook and Post-message webhook configuration pages, you can configure the webhook timeout length from a minimum of 1 second to a maximum of 30 seconds.

Rich response types

Your assistant can now send responses that include elements such as audio, video, or embedded iframe content.

Analytics overview change

To improve reliability, the Values column has been removed from Top entities on the Analytics Overview page. Top Entities continues to provide counts of entity types.

Dialog skill "Try it out" improvements

For dialog skills, the Try it out pane now uses the React UI framework similar to the rest of the Watson Assistant user interface. You shouldn't see any change in behavior or functionality. As a part of the update, dialog skill error handling has been improved within the "Try it out" pane.

Disambiguation feature updates

The dialog skill disambiguation feature now includes improved features:

• Increased control: The frequency and depth of disambiguation can now be controlled by using the sensitivity parameter in the workspace API. There are 5 levels of sensitivity:
  • high
  • medium_high
  • medium
  • medium_low
  • low

  The default (auto) is medium_high if this option is not set.

• More predictable: The new disambiguation feature is more stable and predictable. The choices shown may sometimes vary slightly to enable learning and analytics, but the order and depth of disambiguation is largely stable.

These new features may affect various metrics, such as disambiguation rate and click rates, as well as influence conversation-level key performance indicators such as containment.

If the new disambiguation algorithm works differently than expected for your assistant, you can adjust it using the sensitivity parameter in the update workspace API.

The 4.0.6 release of Watson Discovery includes the following features and updates:

Multitenancy is now supported

An administrator can now create up to 10 instances of the Discovery service per deployment, which means that more teams can work on discrete Discovery projects at the same time.

Simpler installation and management of custom connectors

The manage_custom_crawler.sh script was improved to make it easier for you to install and manage your custom connectors in a multitenant environment. For details, see Installing a custom Cloud Pak for Data connector.

For more information about changes in Watson Discovery, see the Watson Discovery release notes.

The 4.0.6 release of Watson Knowledge Catalog includes the following features and updates:

Access more data with new connection types

In Watson Knowledge Catalog, you can now work with data from these data sources:

• Generic S3 (Use to connect to a storage service that is compatible with the Amazon S3 API)
• Exasol

Data Scientist role has Access governance artifacts permission

With the Access governance artifacts permission, data scientists can see the details of governance artifacts that are assigned to assets to better understand the data. For details, see Predefined roles and permissions.

Keeping catalog names unique

When you create a catalog in the Create a catalog page, you must now use a unique name. Unique catalog names will avoid ambiguity problems and sync errors. If you need to use a duplicate name for a catalog, use the API to rename or create a catalog.

Generating reports on Watson Knowledge Catalog data

Now you can get insights into your catalogs, projects, and governance artifacts by setting up reporting for Watson Knowledge Catalog. The data is sent to an external database where you can run SQL queries to generate reports. For details on configuring the reporting, see Reporting on Watson Knowledge Catalog data.

Enhanced linguistic matching in quick scan

In quick scan, term assignment based on linguistic name matching now provides better results because the longest common substrings of term and asset names are considered.

Version 4.0.6 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

The 4.0.6 release of Watson Machine Learning includes the following features and updates:

Edit hardware specifications for online deployments

You can now update the hardware specification for some online deployments to allocate computing resources to the deployment. For details, see Creating an online deployment.

Support for new data sources for training and deploying AutoAI experiments

Expanded support for connecting to data in popular databases, including Datastax and Exasol, for training and deployment. For details, see:

• AutoAI overview for training sources
• Batch deployment details for deployment sources

Use the latest, more secure frameworks for machine learning

To provide you with more secure frameworks, older software specifications based on Python 3.7 are deprecated and new specifications built on Python 3.9 are available. Additionally, Spark 2.4 is removed, with full support available for Spark 3.0. For details on additions, removals, and deprecations, see Software specifications.

Use storage volumes with your Shiny app deployments

You can now access log files and large data files of Shiny apps with default auto mounting of storage volume connections you created in a project or space. Users can now read and write files from the deployed Shiny app. Cloud Pak for Data also supports disabling auto mount and setting individual mount access via the API. For details, see Creating Shiny app deployments.

Updates to Federated Learning frameworks and API

• The Federated Learning module is now part of the Python client for Watson Machine Learning. All API functions that contain ibmfl will be removed in a future release. Update the Watson Machine Learning module and the party connector script to the latest version.
• Scikit-learn 1.0.2 is now a supported framework for Federated Learning training.
• Python 3.7 is deprecated and will be removed in a future release. Update to Python 3.8 to continue using Federated Learning. Python 3.9 is not supported for training but is supported for deploying models that use the Python 3.9 software spec. For details, see Frameworks and Python version compatibility.

The 2.3.6 release of Watson Machine Learning Accelerator includes the following features and updates:

Use the latest deep learning frameworks

Watson Machine Learning Accelerator now supports the following deep learning libraries based on Python 3.9:

• TensorFlow 2.7.0
• PyTorch 1.10.1
• NVIDIA CUDA Toolkit 11.2.2.

Support for Python 3.9

When you install Watson Machine Learning Accelerator, the service now uses Python 3.9.7 for running deep learning frameworks.

If you are upgrading Watson Machine Learning Accelerator, make sure to update and test your models to use the latest supported frameworks for Python 3.9.7.

Version 2.3.6 of the Watson Machine Learning Accelerator service includes various fixes.

The 4.0.6 release of Watson OpenScale includes the following features and updates:

Expanded authentication options for custom machine learning engines

In addition to basic authentication, you can now authenticate by using API keys if you have custom machine learning engines.

New usability and performance features

Support has been added through the UI and API for these new features:

• setting a new database password through the UI
• defining multiple metric threshold for Quality monitor
• defining threshold for data accuracy in Drift monitor
• publishing OpenScale metrics to OpenPages objects of type "deployment"
• assigning group level access permissions to an OpenScale instance
• allowing users to specify max_records to process during Quality and Drift evaluation
• processing large datasets in feedback/quality monitoring
• configuring a batch deployed python function as a Custom Metric Provider

Version 4.0.6 of the Watson Speech to Text service includes various fixes.

The 4.0.6 release of Watson Studio includes the following features and updates:

Access more data with new connection types

In Watson Studio, you can now work with data from these data sources:

• Generic S3 (Use to connect to a storage service that is compatible with the Amazon S3 API)
• Exasol

Support for Python 3.9

You can now use environments with Python 3.9 when working on notebooks and scripts. The Python 3.9 environment is based on the IBM Runtimes 22.1 release. For information on the IBM Runtimes 22.1 release and the included environments for Python 3.9, see Environments.

Deprecation notice for Python 3.7

Python 3.7 is deprecated and will be removed in an upcoming release. Start using environments with a newer version of Python, which contains newer versions of popular machine learning libraries.

Version 4.0.6 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

The 4.0.6 release of Watson Studio Runtimes includes the following features and updates:

Support for Python 3.9

You can now use environments with Python 3.9 when working on notebooks and scripts. All Python 3.9 environments are based on the IBM Runtimes 22.1 release. Python 3.9 can be used in notebooks and JupyterLab and in combination with Spark 3.0. For information on the IBM Runtimes 22.1 release and the included environments for Python 3.9, see Environments.

Support for Python 3.9 for GPU environments

To use GPU computing power with Python 3.9, you can install the Jupyter Notebooks with Python 3.9 for GPU service. For details, see Installing Jupyter Notebooks with Python 3.9 for GPU.

Manual installation and uninstallation of Python 3.7

Python 3.7 is deprecated and will be removed in an upcoming release.

If you are installing Watson Studio Runtimes for the first time, and you need Python 3.7, you have to install Python 3.7 manually. For details, see Installing Jupyter Notebooks for Python 3.7.

If you upgrade from a previous release of Watson Studio Runtimes, your Python 3.7 and Python 3.8 version are updated with any included security fixes and not removed. You can manually uninstall Python 3.7 if you don't want to use it. For details, see Uninstalling Jupyter Notebooks for Python 3.7.

Version 4.0.6 of the Watson Text to Speech service includes various fixes.

Refresh 5 of Version 4.0 of the Cloud Pak for Data platform includes various fixes.

Version 4.0.5 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

Version 4.0.5 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

Version 4.0.5 of the Cognos Analytics service includes various fixes.

Version 4.0.5 of the Cognos Dashboards service includes various fixes.

Version 4.0.5 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

Version 4.0.5 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

Version 1.7.5 of the Data Virtualization service includes various fixes.

For details, see What's new and changed in Data Virtualization.

Version 4.0.5 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

Version 4.0.7 of the Db2 service includes various fixes.

The 7.2.5 release of Db2 Big SQL includes the following features and updates:

Hadoop cluster support

Db2 Big SQL Version 7.2.5 can connect to Hadoop clusters on Cloudera Data Platform (CDP) Private Cloud Base 7.1.7 only.

Version 7.2.5 of the Db2 Big SQL service includes various fixes.

Version 2.0.5 of the Db2 Data Gate service includes various fixes.

The 4.0.5 release of Db2 Data Management Console includes the following features and updates:

Support for Power hardware

You can now install Db2 Data Management Console on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 or Power 10 hardware. However, the service does not take advantage of Power 10 optimizations.

Version 4.0.5 of the Db2 Data Management Console service includes various fixes.

For details, see What's new and changed in Db2 Data Management Console.

Version 4.0.7 of the Db2 Warehouse service includes various fixes.

Version 4.0.5 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

Version 4.0.5 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

Version 1.1.1750 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

The 4.0.2 release of Informix includes the following features and updates:

New version of Informix

This release of the Informix service runs Informix Version 14.10.FC7W1.

Version 4.0.2 of the Informix service includes various fixes.

Version 8.204.2 of the OpenPages service includes various fixes.

Version 4.0.5 of the Planning Analytics service includes various fixes.

Version 1.0.3 of the Product Master service includes various fixes.

For details, see What's new and changed in Product Master.

Version 4.0.5 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

Version 4.0.5 of the Watson Assistant service includes various fixes.

Version 4.0.5 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

Version 4.0.5 of the Watson Knowledge Studio service includes various fixes.

Version 4.0.5 of the Watson Machine Learning service includes various fixes.

The 2.3.5 release of Watson Machine Learning Accelerator includes the following features and updates:

Accessing the Watson Machine Learning Accelerator service console

The URL to the Watson Machine Learning Accelerator console was removed from the Instances page.

For details on accessing the console, see Deep learning with Watson Machine Learning Accelerator.

Version 2.3.5 of the Watson Machine Learning Accelerator service includes various fixes.

Version 4.0.5 of the Watson OpenScale service includes various fixes.

The 4.0.5 release of Watson Speech to Text includes features and updates.

For a list of new features in Watson Speech to Text, see the Watson Speech to Text release notes for IBM Cloud Pak for Data.

Version 4.0.5 of the Watson Speech to Text service includes various fixes.

For details, see Watson Speech to Text release notes for IBM Cloud Pak for Data.

Version 4.0.5 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

The 4.0.5 release of Watson Text to Speech includes features and updates.

For a list of new features in Watson Text to Speech, see the Watson Text to Speech release notes for IBM Cloud Pak for Data.

Version 4.0.5 of the Watson Text to Speech service includes various fixes.

For details, see the Watson Text to Speech release notes for IBM Cloud Pak for Data.

New method for creating the route to the platform

Cloud Pak for Data now offers two options to create a custom route to the platform. You can use the options separately or in tandem, depending on your needs.

For details, see Changing the route to the platform.

New option to set the install plan on automatically installed operators

Some Cloud Pak for Data services have dependencies that are automatically installed when you install the service. For example, when you install Watson Studio, Watson Studio automatically installs the following dependencies if they are not already available:

• Common core services
• Data Refinery

Previously, the operators for these dependencies were created with the automatic install plan (installPlanApproval: Automatic). Starting in Refresh 4 of Version 4.0, you can specify which install plan these operators use.

For details, see Specifying the install plan for operators that are automatically installed by Operand Deployment Lifecycle Manager.

Refresh 4 of Version 4.0 of the Cloud Pak for Data platform includes various fixes.

Version 4.0.4 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

The 1.3.1 release of the scheduling service includes the following features and updates:

The scheduling service has a higher priority

The scheduling service pods now use priority 2000001000 and priority class system-node-critical. This change prevents the cluster from stopping the scheduling service when there are limited resources.

Version 1.3.1 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

Version 4.0.4 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

The 4.0.4 release of Cognos Analytics includes the following features and updates:

New version of Cognos Analytics

This release of the Cognos Analytics service runs Cognos Analytics Version 11.1.7 FP4.

Version 4.0.4 of the Cognos Analytics service includes various fixes.

Version 4.0.4 of the Cognos Dashboards service includes various fixes.

Version 4.0.4 of the Data Privacy service includes various fixes.

For details, see What's new and changed in Data Privacy.

Version 4.0.4 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

The 4.0.4 release of DataStage includes the following features and updates:

New stage

DataStage now supports the Excel stage to read data from or write data to Excel files.

For details, see Excel stage.

New connectors

DataStage now supports the following additional connectors:

• Amazon RDS for MySQL
• Databases for MongoDB
• MariaDB
• MongoDB

Additionally, the ODBC connection now includes the Text data source.

For details, see DataStage connectors.

Enhancements to the Address Verification (AVI) stage

The AVI stage now includes the following enhancements:

Reverse geocoding

You can provide longitude and latitude coordinates to get an estimated address.

Coding Accuracy Support System (CASS) mode

The AVI stage applies CASS rules as defined by the United States Postal Service® (USPS).

CASS rules are used to correct and standardize addresses. The rules add missing address information, such as ZIP codes, cities, and states, to ensure the address is complete.

For more information on the AVI stage, see Address Verification stage.

Project asset support

You can now use the asset browser connector in DataStage to add files of type .csv, .txt, .xlsx, .json, and .xml directly to the canvas of a DataStage flow. When you add these files in this way, DataStage automatically adds the files to the canvas in the form of the appropriate stage or connector, as shown in the following list:

• .csv and .txt: Sequential file connector
• .xlsx: Excel stage
• .json and .xml: Hierarchical stage

Multiple conductor nodes

You can optionally enable dynamic workload management to use multiple conductor nodes to scale DataStage horizontally. Horizontal scaling provides better workload balancing and availability.

For details, see Enabling multiple conductor (PXRuntime) pods in DataStage.

Version 4.0.4 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

The 4.0.6 release of Db2 includes the following features and updates:

High-speed data replication

High-speed data replication capability for Db2 was announced and is available in Cloud Pak for Data 4.0.4 and 4.0.3. Replication supports continuous availability, data locality, and compliance and regulatory requirements. The replication feature in Db2 provides the ability to fail over and fallback during planned and unplanned outages, with low latency and high-volume throughput. You can protect applications for continuous availability by replicating transactions on the tables that are used by these applications. You can also keep active copies of data in proximity to users, for example allowing east coast clients to access data from an east coast data center, and west coast clients from a west coast data center. For details, see Replication.

New version of Db2

This release of the Db2 service runs Db2 Version 11.5.7.0.

Version 4.0.6 of the Db2 service includes various fixes.

Version 2.0.4 of the Db2 Data Gate service includes various security fixes.

The 4.0.6 release of Db2 Warehouse includes the following features and updates:

High-speed data replication

High-speed data replication capability for Db2 Warehouse was announced and is available in Cloud Pak for Data 4.0.4 and 4.0.3. Replication supports continuous availability, data locality, and compliance and regulatory requirements. The replication feature in Db2 Warehouse, which is limited to SMP environments, provides the ability to fail over and fallback during planned and unplanned outages, with low latency and high-volume throughput. You can protect applications for continuous availability by replicating transactions on the tables that are used by these applications. You can also keep active copies of data in proximity to users, for example allowing east coast clients to access data from an east coast data center, and west coast clients from a west coast data center. For details, see Replication.

New version of Db2 Warehouse

This release of the Db2 Warehouse service runs Db2 Warehouse Version 11.5.7.0.

Version 4.0.6 of the Db2 Warehouse service includes various fixes.

The 4.0.4 release of Decision Optimization includes the following features and updates:

Support for Power 9 hardware

You can now install Decision Optimization on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

The 4.0.4 release of Execution Engine for Apache Hadoop includes the following features and updates:

Support for Power 9 hardware

You can now install Execution Engine for Apache Hadoop on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

The 1.1.167 release of IBM Match 360 includes the following features and updates:

More control over matching

This release provides more control over the matching algorithm by enabling data engineers to select custom attributes and fields for matching consideration.

For details, see Setting up your algorithm and matching your data.

Version 1.1.1670 of the IBM Match 360 service includes various fixes.

For details, see What's new and changed in IBM Match 360.

The 4.0.1 release of Informix includes the following features and updates:

New version of Informix

This release of the Informix service runs Informix Version 14.10.FC7.

Support for multiple editions

This release of Informix supports the following editions:

• Developer Edition
• Workgroup Edition
• Enterprise Edition
• Advanced Enterprise Edition

Version 4.0.1 of the Informix service includes various fixes.

For details, see Fix list for Informix Server 14.10.FC7 release.

Versions 4.2.6 and 4.4.0 of the MongoDB service include various fixes.

Version 8.204.1 of the OpenPages service includes various fixes.

The 4.0.4 release of Planning Analytics includes the following features and updates:

Support for upgrade

You can upgrade now Planning Analytics from the following Cloud Pak for Data releases:

• Cloud Pak for Data Version 3.5 to Refresh 4 of Version 4.0.

Version 4.0.4 of the Planning Analytics service includes various fixes.

The 4.0.4 release of RStudio Server with R 3.6 includes the following features and updates:

Support for Power 9 hardware

You can now install RStudio Server with R 3.6 on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the RStudio Server with R 3.6 service includes various fixes.

For details, see What's new and changed in RStudio Server with R 3.6.

The 4.0.4 release of SPSS Modeler includes the following features and updates:

Support for Power 9 hardware

You can now install SPSS Modeler on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the SPSS Modeler service includes various fixes.

For details, see What's new and changed in SPSS Modeler.

A new refresh of Voice Gateway is available on Cloud Pak for Data Version 4.0. This refresh includes various fixes.

Version 4.0.4 of the Watson Assistant service includes critical security fixes.

Version 4.0.4 of the Watson Discovery service includes various fixes.

Version 4.0.4 of the Watson Knowledge Catalog service includes various fixes.

For details, see What's new and changed in Watson Knowledge Catalog.

Version 4.0.4 of the Watson Knowledge Studio service includes various fixes.

The 4.0.4 release of Watson Machine Learning includes the following features and updates:

Support for Power 9 hardware

You can now install Watson Machine Learning on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the Watson Machine Learning service includes various fixes.

For details, see What's new and changed in Watson Machine Learning.

Version 2.3.4 of the Watson Machine Learning Accelerator service includes various fixes.

The 4.0.4 release of Watson OpenScale includes the following features and updates:

Support on s390x hardware

You can install Watson OpenScale on Red Hat OpenShift Container Platform Version 4.8 clusters running on s390x hardware (IBM Z and LinuxONE.

Version 4.0.4 of the Watson OpenScale service includes various fixes.

The 4.0.4 release of Watson Speech to Text includes features and updates.

For a list of new features in Watson Speech to Text, see the Watson Speech to Text release notes for IBM Cloud Pak for Data.

Version 4.0.4 of the Watson Speech to Text service includes various fixes.

The 4.0.4 release of Watson Studio includes the following features and updates:

Support for Power 9 hardware

You can now install Watson Studio on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the Watson Studio service includes various fixes.

For details, see What's new and changed in Watson Studio.

The4.0.4 release of Watson Studio Runtimes includes the following features and updates:

Support for Power 9 hardware

You can now install Watson Studio Runtimes on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 9 hardware.

Version 4.0.4 of the Watson Studio Runtimes service includes various fixes.

For details, see What's new and changed in Watson Studio Runtimes.

The 4.0.4 release of Watson Text to Speech includes features and updates.

For a list of new features in Watson Text to Speech, see the Watson Text to Speech release notes for IBM Cloud Pak for Data.

Version 4.0.4 of the Watson Text to Speech service includes various fixes.

Updated guidance for installing and upgrading

If you are pulling images directly from the IBM Entitled Registry, it is strongly recommended that you create catalog sources that pull specific versions of images from the registry, rather than using the IBM Operator Catalog.

The IBM Operator Catalog automatically updates the images in your environment, so it is recommended only for short-term installations, such as proof-of-concept deployments. In production and test environments, where predictability and stability are important, you can create catalogs sources that pull a specific version of the software images from the IBM Entitled Registry.

For details, see:

• Installations: Creating catalog sources
• Upgrades from Version 3.5, see Creating catalog sources
• Upgrades from Version 4.0.x, see Updating catalog sources

Managing secrets and vaults

Cloud Pak for Data services use sensitive information, such as database access credentials, API keys, and authentication tokens. You can use secrets and vaults to store this sensitive data.

By default, Cloud Pak for Data includes an internal vault that is accessible through the Credentials and Secrets API. However, a Red Hat OpenShift Container Platform project administrator can enable the vaults interface, which enables users to:

• Add secrets to the internal vault from the web client
• Use secrets from an external vault, such as CyberArk or HashiCorp

You can use these secrets when you create connections. However, not all services can use connections that use secrets. The following services support connections that use secrets from vaults:

• Data Virtualization
• Decision Optimization
• Execution Engine for Apache Hadoop
• OpenPages
• Watson Knowledge Catalog
• Watson Studio Runtimes

For information on enabling Cloud Pak for Data to use vaults, see Configuring vault usage.

For information on adding secrets to the internal vault or connecting to external vaults, see Managing secrets and vaults.

Support for Power 10 hardware

You can now install Cloud Pak for Data on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 10 hardware. However, the platform does not take advantage of Power 10 optimizations.

Refresh 3 of Version 4.0 of the Cloud Pak for Data platform includes various fixes.

Version 10.0.2 of theCloud Pak for Data command-line interface includes various fixes.

Version 4.0.3 of the common core services includes various fixes.

For details, see What's new and changed in the common core services.

The 1.3.0 release of the scheduling service includes the following features and updates:

Architectural changes

This release includes several architectural changes:

• The scheduling service no longer requires a persistent volume to store data
• The scheduling service no longer uses agent pods
• Prometheus metrics have been moved to a dedicated deployment, named ibm-cpd-scheduler-metric

Improvements to liveness checks

If either of the following components are unresponsive for too long, they are automatically restarted:

• Scheduler
• Parallel Job controller

Support for Power 10 hardware

You can now install the scheduling service on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 10 hardware. However, the service does not take advantage of Power 10 optimizations.

Version 1.3.0 of the scheduling service includes various fixes.

For details, see What's new and changed in the scheduling service.

The 4.0.3 release of Analytics Engine Powered by Apache Spark includes the following features and updates:

Support on Power 9 hardware

You can install Analytics Engine Powered by Apache Spark on a Red Hat OpenShift Container Platform Version 4.8 cluster running on Power 9 hardware.

Support on s390x hardware

You can run Spark applications in Analytics Engine Powered by Apache Spark on Red Hat OpenShift Container Platform Version 4.8 running on IBM Z and LinuxONE. However, deployments on this hardware support a limited set of features.

For a list of the features that are available on IBM Z and LinuxONE, see Capabilities on IBM Z.

Version 4.0.3 of the Analytics Engine Powered by Apache Spark includes various fixes.

For details, see What's new and changed in Analytics Engine Powered by Apache Spark.

Version 4.0.3 of the Cognos Analytics service includes various fixes.

The 4.0.3 release of Cognos Dashboards includes the following features and updates:

Support for Power 10 hardware

You can now install Cognos Dashboards on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 10 hardware. However, the service does not take advantage of Power 10 optimizations.

Version 4.0.3 of the Cognos Dashboards service includes various fixes.

The Data Privacy service is now available on Cloud Pak for Data Version 4.0

Data Privacy enables you to protect sensitive data with features such as:

Advanced masking options

The service includes masking options such as enhanced format-preservation, one-way tokenization, reversible encryption, and relationship integrity in data protection rules. For details, see Advanced data masking.

Define the data that you want to mask

When you create a masking flow, you can choose whether you want to mask a complete copy of the data or a subset of the data. For details, see Creating masking flows.

Self-service access to masked data

You can add masked data to catalogs and collaborative project to enable controlled access to the data For details, see Masking data with Data Privacy.

The 4.0.3 release of Data Refinery includes the following features and updates:

New user interface for Data Refinery operation steps

Data Refinery introduces a new user interface that makes it easier to work with your data. The Steps panel is consolidated with the operations.

For details, see Refining data.

Reorganized categories for Data Refinery GUI operations

The Data Refinery GUI operations are reorganized into the following categories:

• CLEANSE
• COMPUTE
• ORGANIZE
• NATURAL LANGUAGE

For details, see GUI operations.

Performance enhancements for GUI operations in Data Refinery flows that use large data assets

The following GUI operations have performance enhancements.

• Convert column type to Date or to Timestamp

  This also applies if the flow uses the Convert column type operation as the automatic first step in the flow.

• Remove stop words
• Replace substring
• Text > Pad characters
• Text > Substring

Important: If you have existing Data Refinery flows that use these GUI operations, you must update the flows. To update a flow, open it, save it, and run a job for the flow.

For details, see Managing Data Refinery flows.

Version 4.0.3 of the Data Refinery service includes various fixes.

For details, see What's new and changed in Data Refinery.

Support for upgrade

You can upgrade Data Virtualization from the following Cloud Pak for Data releases:

• Cloud Pak for Data Version 3.5.x (Data Virtualization 1.5.0)
• Cloud Pak for Data Version 4.0.1 (Data Virtualization 1.7.1)

Support for vaults

Data Virtualization supports integration with an external vault where your secrets are stored. For more information, see Managing secrets and vaults.

You can use vaults and secrets to store and access credentials securely, and then use them to connect to data sources. For more information, see Using secrets from vaults in connections.

Note: You cannot use credentials that are stored in a vault to connect to data sources that are accessed by using remote connectors in Data Virtualization.

Enhanced data masking

The substitute data masking action in Data Virtualization now supports CHAR and VARCHAR columns of any length.

Integers and decimals now support substitution by displaying a numeric hash of the original value. The resulting hash has from two digits up to the same number of digits as the original value.

For more information, see Masking virtual data.

Tech preview This is a technology preview and is not supported for use in production environments.

Improved handling of String and Binary types in Hive data sources

String and Binary types in Hive data sources now resolve to VARCHAR(32672) and VARBINARY(32672) by default, instead of CLOB and BLOB. A Data Virtualization Admin can configure HIVE_MAX_STRING_SIZE and HIVE_MAX_BINARY_SIZE. This update optimizes SQL performance for Hive data sources. For more information, see Supported data sources.

Version 1.7.3 of the Data Virtualization service includes various fixes.

For details, see What's new and changed in Data Virtualization.

The 4.0.3 release of DataStage includes the following features and updates:

New stages

DataStage includes new stages, which give you more tools to process your data:

• Address Verification
• Hierarchical (XML)
• Java™ Integration
• Pivot Enterprise
• Surrogate Key Generator

For more information, see DataStage stages.

New connectors

DataStage includes new connectors:

• Google Cloud Pub/Sub
• MySQL

For more information, see DataStage connectors.

Reusable components

You can create components that you use in projects and in DataStage flows. You create these components in a project, outside of a DataStage flow, which gives you the flexibility to reuse the components in separate places. The components are stored as assets in your project. You can create the following components:

• Data definitions
• Parameter sets
• Subflows

For more information, see the following topics:

• Defining data definitions
• Creating and using parameters and parameter sets
• Subflows

Version 4.0.3 of the DataStage service includes various fixes.

For details, see What's new and changed in DataStage.

The 4.0.5 release of Db2 includes the following features and updates:

Support for Power 10 hardware

You can now install Db2 on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 10 hardware. However, the service does not take advantage of Power 10 optimizations.

Version 4.0.5 of the Db2 service includes various fixes.

The 7.2.3 release of Db2 Big SQL includes the following features and updates:

Ability to terminate user applications

Users can now terminate their own applications through the Db2 Data Management Console. Previously, only administrators could terminate user applications. For details, see Working with Db2 Big SQL databases.

Improved user experience for Active Directory

The Db2 Big SQL user interface provides an improved experience when you are connecting to a Hadoop cluster that uses Active Directory as a Kerberos server. For details, see Setting up a connection to a remote data source.

Improved user experience when upgrading on Cloud Pak for Data Version 4.0

If you are running Db2 Big SQL on Cloud Pak for Data Version 4.0, the process to upgrade to the latest refresh of Db2 Big SQL has been streamlined. For details, see Upgrading Db2 Big SQL to a newer refresh on Version 4.0.

Version 7.2.3 of the Db2 Big SQL service includes various fixes.

The 2.0.3 release of Db2 Data Gate includes the following features and updates:

Read request delay

SQL read requests can be postponed to wait for synchronization processes to finish. This ensures that a request fetches current and recently committed data. For more information, see Making queries wait for synchronization updates.

Schema change support

Structural changes to a Db2 for z/OS® table, such as adding a column, are reflected in the corresponding Db2 Data Gate table. This eliminates the need to remove, re-add, and reload the Db2 Data Gate table after a structural change.

Distributed deployment

You are no longer required to create the Db2 Data Gate instance on the same OpenShift node as the target database. You can choose whether the instances co-exist or whether they are distributed. A distributed installation ensures that Db2 Data Gate and the target database do not compete for CPU and memory resources. For more information, see Creating a Db2 Data Gate instance.

Version 2.0.3 of the Db2 Data Gate service includes various fixes.

The 4.0.3 release of Db2 Data Management Console includes the following features and updates:

Support for index management

Db2 Data Management Console supports index management, which enables a user to alter, drop, generate DDL, and view the dependencies of database objects.

High-speed replication capability

You can now use the Db2 Data Management Console to set up high-speed data replication on the Db2 and Db2 Warehouse services on Cloud Pak for Data. Use the Db2 Data Management Console to activate replication, create replication sets, and perform operational tasks such as starting and stopping. You can also use the graphical charts to monitor replication latency and throughput.

Version 4.0.3 of the Db2 Data Management Console service includes various fixes.

For details, see What's new and changed in Db2 Data Management Console.

The 4.0.5 release of Db2 Warehouse includes the following features and updates:

Support for Power 10 hardware

You can now install Db2 Warehouse on Red Hat OpenShift Container Platform Version 4.8 clusters running on Power 10 hardware. However, the service does not take advantage of Power 10 optimizations.

The 4.0.3 release of Decision Optimization includes the following features and updates:

CPLEX® and CPO models support in Decision Optimization experiments (Watson Studio)

You can now import and solve CPLEX and CPO files in Decision Optimization Experiments. For details, see Build model view.

Decision Optimization experiment UI improvements (Watson Studio)

• You can more easily select how to formulate your model in the Decision Optimization Experiment UI. For details, see Build model.
• You can now zoom in on the solution graph in the Explore Solution view.

Deprecation notice for CPLEX 12.9

The CPLEX 12.9 model type is deprecated in Watson Studio andWatson Machine Learning and will be removed in an upcoming release. Migrate to the latest version, CPLEX 20.1.

For details on Decision Optimization model types, see Model deployment.

Version 4.0.3 of the Decision Optimization service includes various fixes.

For details, see What's new and changed in Decision Optimization.

In this refresh, the PostgreSQL operator is updated to Version 1.9.2.

Version 4.0.3 of the Execution Engine for Apache Hadoop service includes various fixes.

For details, see What's new and changed in Execution Engine for Apache Hadoop.

The 1.1.134 release of IBM Match 360 includes the following features and updates:

Backup and restore

IBM Match 360 now supports restoring a backed up instance to a different cluster. For details, see Backing up and restoring your deployment.

Performance improvements for matching

The IBM Match 360 matching process has been improved to reduce the time it takes to run the matching algorithm.

Support for Spark 3

IBM Match 360 now uses Spark 3 as its processing engine.