What's new and changed in Data Refinery

Data Refinery updates can include new features, bug fixes, and security updates. Updates are listed in reverse chronological order so that the latest release is at the beginning of the topic.

You can see a list of the new features for the platform and all of the services at What's new in IBM Cloud Pak for Data?

Installing or upgrading Data Refinery

Ready to install or upgrade Data Refinery?

Related documentation:

Cloud Pak for Data Version 4.5.3

A new version of Data Refinery was released in October 2022 with Cloud Pak for Data 4.5.3.

Operand version: 4.5.3

This release includes the following changes:

Security fixes

CVE-2022-30580, CVE-2022-25168, CVE-2022-1927, CVE-2022-1897, CVE-2022-1785, CVE-2022-1586

CVE-2019-20916

Cloud Pak for Data Version 4.5.2

A new version of Data Refinery was released in August 2022 with Cloud Pak for Data 4.5.2.

Operand version: 4.5.2

This release includes the following changes:

Customer-reported issues fixed in this release
DT180666: HTTPS protocol and ports 443 on deployments error
Security fixes

This release includes fixes for the following security issues:

CVE-2016-5131, CVE-2016-4658

CVE-2017-9782, CVE-2017-9050, CVE-2017-9049, CVE-2017-9048, CVE-2017-9047, CVE-2017-7375, CVE-2017-6519, CVE-2017-5505, CVE-2017-5504, CVE-2017-5503, CVE-2017-5499, CVE-2017-18258, CVE-2017-17095, CVE-2017-15713, CVE-2017-15412, CVE-2017-14501, CVE-2017-14166, CVE-2017-13745, CVE-2017-0663

CVE-2018-9252, CVE-2018-9055, CVE-2018-8009, CVE-2018-6872, CVE-2018-5360, CVE-2018-20846, CVE-2018-20839, CVE-2018-20786, CVE-2018-20673, CVE-2018-20671, CVE-2018-20657, CVE-2018-20651, CVE-2018-20623, CVE-2018-20622, CVE-2018-20570, CVE-2018-20406, CVE-2018-20002, CVE-2018-19932, CVE-2018-19876, CVE-2018-19543, CVE-2018-19542, CVE-2018-19541, CVE-2018-19540, CVE-2018-19539, CVE-2018-19478, CVE-2018-19211, CVE-2018-19210, CVE-2018-19139, CVE-2018-18873, CVE-2018-18701, CVE-2018-18700, CVE-2018-18607, CVE-2018-18606, CVE-2018-18605, CVE-2018-18484, CVE-2018-18483, CVE-2018-18309, CVE-2018-18064, CVE-2018-18020, CVE-2018-17985, CVE-2018-17794, CVE-2018-17360, CVE-2018-17101, CVE-2018-16435, CVE-2018-16428, CVE-2018-16335, CVE-2018-15919, CVE-2018-15209, CVE-2018-1302, CVE-2018-12934, CVE-2018-12699, CVE-2018-12698, CVE-2018-12697, CVE-2018-12641, CVE-2018-1121, CVE-2018-10801, CVE-2018-10779, CVE-2018-1000880, CVE-2018-1000879, CVE-2018-1000876, CVE-2018-1000654, CVE-2018-1000021

CVE-2019-9937, CVE-2019-9936, CVE-2019-9923, CVE-2019-9674, CVE-2019-9543, CVE-2019-9077, CVE-2019-9075, CVE-2019-9074, CVE-2019-9071, CVE-2019-8906, CVE-2019-8905, CVE-2019-6462, CVE-2019-6461, CVE-2019-6128, CVE-2019-6110, CVE-2019-3862, CVE-2019-3861, CVE-2019-3860, CVE-2019-3859, CVE-2019-3858, CVE-2019-25059, CVE-2019-2201, CVE-2019-19244, CVE-2019-17567, CVE-2019-17543, CVE-2019-17498, CVE-2019-16866, CVE-2019-16777, CVE-2019-16775, CVE-2019-14274, CVE-2019-14250, CVE-2019-13115, CVE-2019-12972, CVE-2019-12900, CVE-2019-11026

CVE-2020-9492, CVE-2020-8492, CVE-2020-7608, CVE-2020-35512, CVE-2020-35507, CVE-2020-35496, CVE-2020-35495, CVE-2020-35494, CVE-2020-35493, CVE-2020-27748, CVE-2020-21674, CVE-2020-17049, CVE-2020-16156, CVE-2020-15095, CVE-2020-14422, CVE-2020-13790, CVE-2020-12413, CVE-2020-12401

CVE-2021-46822, CVE-2021-46195, CVE-2021-45949, CVE-2021-45944, CVE-2021-45941, CVE-2021-45940, CVE-2021-45078, CVE-2021-44568, CVE-2021-43618, CVE-2021-4217, CVE-2021-4209, CVE-2021-4166, CVE-2021-4048, CVE-2021-40330, CVE-2021-3997, CVE-2021-3974, CVE-2021-39537, CVE-2021-3927, CVE-2021-38593, CVE-2021-37404, CVE-2021-36978, CVE-2021-3618, CVE-2021-35939, CVE-2021-35938, CVE-2021-35937, CVE-2021-3468, CVE-2021-3467, CVE-2021-3443, CVE-2021-3114, CVE-2021-25317, CVE-2021-24032, CVE-2021-22570, CVE-2021-21300, CVE-2021-20193

CVE-2022-34903, CVE-2022-33980, CVE-2022-33099, CVE-2022-33068, CVE-2022-32208, CVE-2022-32206, CVE-2022-31030, CVE-2022-30556, CVE-2022-30522, CVE-2022-29404, CVE-2022-28506, CVE-2022-27943, CVE-2022-27406, CVE-2022-27405, CVE-2022-27404, CVE-2022-27337, CVE-2022-26612, CVE-2022-25310, CVE-2022-25309, CVE-2022-25308, CVE-2022-25255, CVE-2022-24765, CVE-2022-23943, CVE-2022-23648, CVE-2022-2344, CVE-2022-2343, CVE-2022-2289, CVE-2022-2288, CVE-2022-2287, CVE-2022-2286, CVE-2022-2285, CVE-2022-22844, CVE-2022-2284, CVE-2022-22719, CVE-2022-2257, CVE-2022-2231, CVE-2022-2210, CVE-2022-2208, CVE-2022-2207, CVE-2022-2206, CVE-2022-2183, CVE-2022-2182, CVE-2022-2175, CVE-2022-2129, CVE-2022-2126, CVE-2022-2125, CVE-2022-2124, CVE-2022-2058, CVE-2022-2057, CVE-2022-2056, CVE-2022-1720, CVE-2022-1619, CVE-2022-1355, CVE-2022-1354, CVE-2022-1304, CVE-2022-1122, CVE-2022-0924, CVE-2022-0909, CVE-2022-0908, CVE-2022-0891, CVE-2022-0865, CVE-2022-0562, CVE-2022-0561, CVE-2022-0530, CVE-2022-0529, CVE-2022-0391, CVE-2022-0351, CVE-2022-0235

Cloud Pak for Data Version 4.5.1

A new version of Data Refinery was released in July 2022 with Cloud Pak for Data 4.5.1.

Operand version: 4.5.1

This release includes the following changes:

New features

The 4.5.1 release of Data Refinery includes the following features and updates:

The Default Spark 3.0 & R 3.6 environment is discontinued
If you have any Data Refinery flow jobs set up with the Default Spark 3.0 & R 3.6 environment or a custom environment that uses Spark 3.0, the jobs will fail. Change the environment to one of the following environments:
  • Default Spark 3.2 & R 3.6
  • Default Data Refinery XS
  • A custom environment that does not use Spark 3.0

For information, see Data Refinery environments.

Security fixes

This release includes fixes for the following security issues:

CVE-2021-40528, CVE-2021-44569, CVE-2021-44570, CVE-2021-44571, CVE-2021-44573, CVE-2021-44574, CVE-2021-44575, CVE-2021-44576, CVE-2021-44577

CVE-2022-1271, CVE-2022-1621, CVE-2022-1629, CVE-2022-22576, CVE-2022-25313, CVE-2022-25314, CVE-2022-27774, CVE-2022-27776, CVE-2022-27782, CVE-2022-29824

Cloud Pak for Data Version 4.5.0

A new version of Data Refinery was released in June 2022 with Cloud Pak for Data 4.5.0.

Operand version: 4.5.0

This release includes the following changes:

New features

The 4.5.0 release of Data Refinery includes the following features and updates:

Review and change the data types assigned in the first step of the flow
Data Refinery automatically inserts a first step in the flow to convert any non-string data types that it finds in the data to inferred types. Now you can view these inferred data types and change them, if appropriate.
Edit auto-convert operation

For information about the Convert column type operation, see GUI operations.

View your Data Refinery data in a CSV file without running a Data Refinery flow job
You can now export the data at the current step in your Data Refinery flow to a CSV file without saving or running a Data Refinery flow job. With this enhancement, you can now quickly save and view data that is in progress. For information, see Managing Data Refinery flows.
Control the placement of a new column in a Data Refinery flow
When you use an operation that can create a new column in the Data Refinery flow and you select Create a new column for results, you can now optionally place the new column to the right of the original column.
Option for placing new column

This option is available for the following operations:

  • Calculate
  • Conditional replace
  • Convert column type
  • Convert column value to missing
  • Extract date or time value
  • Math
  • Replace missing values
  • Replace substring
  • Text
  • Tokenize
New step options give you more control of your Data Refinery flow
Data Refinery introduces new options for the steps that give you greater flexibility and control of the Data Refinery flow:
  • Duplicate
  • Insert step before
  • Insert step after

You can access these options from the Steps pane. For information about the actions you can do with steps, see Managing Data Refinery flows.

New Spark 3.2 environment for running Data Refinery flow jobs
You can now select Default Spark 3.2 & R 3.6 when you select an environment for a Data Refinery flow job. The environment includes enhancements from Spark.

The Default Spark 3.0 & R 3.6 environment is deprecated.

For details, see Data Refinery environments.

Bug fixes

This release includes the following fixes:

  • Issue: Cannot run a Data Refinery flow job with certain unsigned data types.

    Resolution: This issue is fixed.

  • Issue: Cannot view visualization charts in Data Refinery after upgrade.

    Resolution: This issue is fixed.

Security fixes

This release includes fixes for the following security issues:

CVE-2001-1268, CVE-2001-1269

CVE-2005-0602

CVE-2016-10739

CVE-2017-12837, CVE-2017-12883, CVE-2017-5932

CVE-2018-1000876, CVE-2018-10372, CVE-2018-10373, CVE-2018-10534, CVE-2018-10535, CVE-2018-12015, CVE-2018-12641, CVE-2018-12697, CVE-2018-12698, CVE-2018-12699, CVE-2018-12934, CVE-2018-13033, CVE-2018-14404, CVE-2018-18311, CVE-2018-18312, CVE-2018-18313, CVE-2018-18314, CVE-2018-19591, CVE-2018-19931, CVE-2018-19932, CVE-2018-20482, CVE-2018-20671, CVE-2018-20796, CVE-2018-25032, CVE-2018-6543, CVE-2018-6759, CVE-2018-6797, CVE-2018-6798, CVE-2018-6872, CVE-2018-6913, CVE-2018-7208, CVE-2018-7568, CVE-2018-7569, CVE-2018-7570, CVE-2018-7642, CVE-2018-7643, CVE-2018-8945, CVE-2018-9138, CVE-2018-9996

CVE-2019-1010204, CVE-2019-11236, CVE-2019-18276, CVE-2019-19126, CVE-2019-19956, CVE-2019-25013, CVE-2019-6488, CVE-2019-7309, CVE-2019-9169, CVE-2019-9192, CVE-2019-9923

CVE-2020-10029, CVE-2020-10543, CVE-2020-10878, CVE-2020-12723, CVE-2020-15366, CVE-2020-1751, CVE-2020-1752, CVE-2020-19131, CVE-2020-27618, CVE-2020-35492, CVE-2020-35493, CVE-2020-35494, CVE-2020-35495, CVE-2020-35496, CVE-2020-35507, CVE-2020-6096

CVE-2021-20193, CVE-2021-20197, CVE-2021-23222, CVE-2021-31879, CVE-2021-3326, CVE-2021-33503, CVE-2021-3487, CVE-2021-3517, CVE-2021-3518, CVE-2021-3537, CVE-2021-3541, CVE-2021-3580, CVE-2021-35942, CVE-2021-3634, CVE-2021-3711, CVE-2021-3712, CVE-2021-37322, CVE-2021-3800, CVE-2021-38604, CVE-2021-4160, CVE-2021-41617, CVE-2021-4189, CVE-2021-42694, CVE-2021-43818, CVE-2021-45078, CVE-2021-45960, CVE-2021-46143

CVE-2022-0778, CVE-2022-1616, CVE-2022-1620, CVE-2022-1622, CVE-2022-1623, CVE-2022-22822, CVE-2022-22823, CVE-2022-22824, CVE-2022-22825, CVE-2022-22826, CVE-2022-22827, CVE-2022-23218, CVE-2022-23219, CVE-2022-23308, CVE-2022-24757, CVE-2022-26280, CVE-2022-29155