What's new and changed in Data Refinery

Data Refinery updates include new features. Updates are listed in reverse chronological order so that the latest release is at the beginning of the topic.

Cloud Pak for Data Version 4.8.3

A new version of Data Refinery was released in February 2024 with Cloud Pak for Data 4.8.3.

This release includes the following changes:

New features
New Spark 3.4 environment for running Data Refinery flow jobs
When you select an environment for a Data Refinery flow job, you can now select Default Spark 3.4 & R 4.2, which includes enhancements from Spark.

The Default Spark 3.3 & R 4.2 environment is deprecated and will be removed in a future update.

Update your Data Refinery flow jobs to use the new Default Spark 3.4 & R 4.2 environment. For more information, see Data Refinery environments.

Cloud Pak for Data Version 4.8.0

A new version of Data Refinery was released in November 2023 with Cloud Pak for Data 4.8.0.

This release includes the following changes:

New features
Control the placement of a new column in the Concatenate operation
You now have two options to specify the position of the new column is created by the Concatenate operation:
  • As the rightmost column in the data set
  • Next to the original column

Previously, the new column was placed at the beginning of the data set.

Important: If you have any existing Data Refinery flows that use the Concatenate operation, edit the flows to specify the new column position for the column. Otherwise, the flows with the operation might fail.

For information about Data Refinery operations, see GUI operations in Data Refinery.