Table of contents

What's new and changed in DataStage

The DataStageĀ® release and subsequent refreshes can include new features, bug fixes, and security updates. Refreshes appear in reverse chronological order, and only the refreshes that contain updates for DataStage are shown.

You can see a list of the new features for the platform and all of the services at What's new in IBM Cloud Pak for Data?

Installing or upgrading DataStage

Ready to install or upgrade DataStage?

DataStage Enterprise
Related documentation:
DataStage Enterprise Plus
Related documentation:

Refresh 5 of Cloud Pak for Data Version 3.5

New versions of DataStage Enterprise and DataStage Enterprise Plus were released in May 2021.

Assembly version: 3.5.5

This release includes the following changes:
Bug fixes
This release includes the following fixes for DataStage Enterprise and DataStage Enterprise Plus:
  • Issue: Apache XMLBeans JAR files need to be updated.

    Resolution: Apache XMLBeans were upgraded from xmlbeans-2.6.0.jar to xmlbeans-3.0.0.jar.

  • Issue: DataStage data transformation jobs that use job parameters are not compiling successfully.

    Resolution: DataStage data transformation jobs now compile successfully.

  • Issue: Job runs are failing with the error Parameter not resolvable. Job parameters that have derivations that include nested parentheses are not resolving and the jobs are failing.

    Resolution: Jobs that include parameter derivations that have nested parentheses run successfully.

  • Issue: The scheduling of some DataStage jobs is not working.

    Resolution: Timezone discrepancies are fixed and you can schedule jobs.

  • Issue: When Db2 is used as both a source and target in a job, and you update the database name in the target, Db2 does not persist the change.

    Resolution: The new database name is successfully updated for both the source and target.

  • Issue: When you add an .xsd file to a library, the file is not added and the OK button is disabled with the error message: Uncaught RangeError: Maximum call stack size exceeded at FileReader.i.onloadend (index_bundle.js:156).

    Resolution: You can add an .xsd file to a library successfully.

Refresh 4 of Cloud Pak for Data Version 3.5

New versions of DataStage Enterprise and DataStage Enterprise Plus were released in March 2021.

Assembly version: 3.5.4

This release includes the following changes:
Bug fixes
This release includes the following fixes for DataStage Enterprise and DataStage Enterprise Plus:
  • Issue: When you open the Hierarchical stage, you receive a NullPointerException message in the systemout.log file.

    Resolution: You can successfully load the Hierarchical stage.

  • Issue: When some jobs with parameter sets are validated, the JSON file is overwritten and every parameter set is set to the same parameters.

    Resolution: Jobs with parameter sets are successfully validated.

  • Issue: It takes a long time to open the Hierarchical stage.

    Resolution: The Hierarchical stage loads immediately.

  • Issue: Jackson-databind jar files need to be updated.

    Resolution: The Jackson-databind jar files are updated.

Refresh 3 of Cloud Pak for Data Version 3.5

New versions of DataStage Enterprise and DataStage Enterprise Plus were released in February 2021.

Assembly version: 3.5.3

This release includes the following changes:
Bug fixes
This release includes the following fixes for DataStage Enterprise and DataStage Enterprise Plus:
  • Issue: When you drag a stage onto the canvas and choose the first job name from the drop-down list, a job is not selected internally. However, the UI displays the job as selected. As a result, the compilation fails and an error is displayed that says the job name is missing.

    Resolution: The first name in the drop-down list is now an empty value, with all the job names listed after the empty value. As a result, any job name that you select from the list is properly set and compile will not fail anymore.

  • Issue: SSL connection failures -- SSL certification files are persistent and available even after pods restart. The environment variable APT_SSL_CONNECTIONS must be set in the directory DSEngine/dsenv to avoid the issue.

    Resolution: SSL connections are successful. Setting the environment variable is not necessary.

  • Issue: When you run a job with the transformer stage, it fails with the error "Failed to load the library".

    Resolution: Jobs with the Transformer stage run successfully.

  • Issue: Compilation process has multiple issues.

    Resolution: Compilation process runs successfully.

  • Issue: Sequence Job: DataStage compiler produces "DSParams value missing" error.

    Resolution: DataStage compiler runs successfully.

  • Issue: Sequence job: Compilation fails on double quotation marks inside execute command.

    Resolution: Compilation is successful on double quotation marks inside execute command.

  • Issue: Job run fails to start and produces the error: "unable to set list-value".

    Resolution: Job runs successfully.

  • Issue: Sequencer Job: Compilation fails for execute command with compile options.

    Resolution: Compilation is successful.

  • Issue: Sequencer job: Compilation fails for routine activity with some built-in routines.

    Resolution: Compilation is successful.

  • Issue: Hierarchical stage: Not able to choose the XML source file. Errors are produced.

    Resolution: XML source file can be chosen successfully.

  • Issue: Unstructure stage: When the Unstructured Data stage is being used, the read option 'Specify the entire data range' is not persisted.

    Resolution: The read option 'Specify the entire data range' is successfully persisted.

  • Issue: Unstructured Data stage: Missing sheet name.

    Resolution: Sheet name is not missing.

  • Issue: Unstructured Data stage: When you use 'Modify' mode for Unstructured Data stage, the columns are not persisted.

    Resolution: When you use 'Modify' mode for Unstructured Data stage, the columns are successfully persisted.

  • Issue: Renaming node for stages like unstructured data target and SCD does not persist.

    Resolution: Renaming node for stages like unstructured data target and SCD persists successfully.

  • Issue: Otherwise/Log toggle field is not persisted when the Transformer stage is edited after the target stage is edited.

    Resolution: Toggle field is persisted successfully.

  • Issue: UI elements: Switch project modal buttons are off. Borders are rounded. Buttons are misaligned with one another.

    Resolution: UI elements display correctly.

  • Issue: The bookmark icon is too far away from the name.

    Resolution: Bookmark icon was moved closer to the name by about 50 px.

  • Issue: Sequencer job: The execute command fails on compilation if the parameter sets are passed in the command property.

    Resolution: The execute command is successful on compilation if the parameter sets are passed in the command property.

Refresh 2 of Cloud Pak for Data Version 3.5

New versions of DataStage Enterprise and DataStage Enterprise Plus were released in January 2021.

Assembly version: 3.5.2

This release includes the following changes:

Bug fixes
This release includes the following fixes for DataStage Enterprise and DataStage Enterprise Plus:
  • Issue: APAR JR63107 - The compiler produces a Stack Overflow error.

    Resolution:The compiler does not produce a Stack Overflow error.

  • Issue: Sequencer job: Job run fails for a Sequencer job that has a start loop with two input links.

    Resolution: Job run is successful for a Sequencer job that has a start loop with two input links.

  • Issue: Cannot upload a keytab file for Spark configuration.

    Resolution: Can upload a keytab file for Spark configuration.

  • Issue: A job that contains a double reference Lookup fails to compile.

    Resolution: A job that contains a double reference Lookup compiles successfully.

  • Issue: Parameter set processing is incorrect in Sequencer job, causing incorrect job results.

    Resolution: Parameter set processing is correct in Sequencer job, and job results are correct.

  • Issue: Hierarchical stage: User is not able to choose the XML source file. Errors are produced.

    Resolution: User is able to successfully choose the XML source file.

Initial release of Cloud Pak for Data Version 3.5

New versions of DataStage Enterprise and DataStage Enterprise Plus were released as part of Cloud Pak for Data Version 3.5.

Assembly version: 3.5.0

This release includes the following changes for DataStage Enterprise and DataStage Enterprise Plus:

New features
Support for additional connectors
You can now connect to the following data sources:
  • Microsoft Azure Data Lake Store
  • Amazon Redshift
  • Unstructured Data
  • SAP Packs
    • A license is required to use SAP Packs in Cloud Pak for Data. SAP Packs require the legacy Windows DataStage Client to design the jobs. Jobs can then be run in Cloud Pak for Data.
    • User documentation is provided with the license for SAP Packs.
    • For more information on SAP Packs, see:

For more details, see Supported connectors.

Additional improvements and updates
  • You can now access DataStage from your projects page. You can create a DataStage project by following the path Projects > All Projects, then creating a new project of type Data transform.
  • Project creation and deletion are now asynchronous. Previously, the DataStage UI was blocked during the time that is required to create or delete a project. Now, you see a notification that says that the request to create or delete the project is submitted. The project appears after the creation or deletion process completes successfully.
  • You can now set up an NFS mount in DataStage pods to pass data files such as CSV and XML between DataStage and source or target systems.
  • You can now use dynamic configuration files without enabling PXRuntime. With this support, the nodes or pods that are used in the job are dynamically decided based on the availability of resources on them at the time of running the job. You can run your jobs by automatically using nodes that have highest resources available, increasing speed and performance.
  • You can change the resource allocation for the number of CPUs and memory to be used in your jobs.
  • Support is provided for SSL/TLS communication with RPC connection by using Nginx as a proxy server. This support provides greater security for connecting the legacy DataStage Designer client to Cloud Pak for Data. You can then use the Designer client to edit jobs in Cloud Pak for Data
  • You can create custom images to support third-party drivers. Custom images have the benefits of being unchangeable after they are built and reliably consistent across different environments. You can also scan the images for vulnerability.
  • You can use a PersistentVolume (PV) to support third-party libraries and drivers.
  • The Operations Console is enabled for stand-alone DataStage installation on Cloud Pak for Data.
  • Non en-US language packs are now supported.
  • Notification with mailx is supported. Notifications can be sent out by mailx after an activity completes in a job sequence.
  • The FileConnector heap size setting and the message handler settings are now persistent and will not be lost if pods are restarted.
  • You can now add parameters and parameter sets in the transformer dialog box.
  • LongVarChar lengths of up to 3,000,000 characters are now supported in the Transformer stage.