What's new and changed in DataStage

The 4.5.3 release of DataStage includes the following features and updates:

Stored procedures for additional connectors

You can now use stored procedures in the following connectors:

• IBM Db2® for z/OS®
• IBM Db2 for i

For more information, see Using stored procedures.

Connect to storage volumes as data sources in DataStage flows

You can now include data from a volume on an NFS server or a persistent volume claim in your DataStage flows. For more information, see Adding connections to analytics projects and Asset browser.

Before-job and after-job subroutines support KSH scripts and bash

You can now use before-job and after-job subroutines to run both bash and KSH scripts. For more information, see Setting up before-job and after-job subroutines.

Share files with Watson™ Studio Pipelines

You can now use a shared storage volume to share files between Watson Studio Pipelines and DataStage. For details, see Sharing storage volumes.

New stage

You can now use the Slowly Changing Dimension stage in your DataStage flows to store and manage current and historical data over time.

For more information, see Slowly Changing Dimension stage.

Simplify setup with the DSStageName server macro

You can add the DSStageName macro to stage properties or in transformer functions. You can use the macro to simplify the setup of DataStage jobs and flows because the macro acts as a DataStage function and outputs data without the need for arguments. When the job compiles, "DSStageName" is replaced with the name of the stage.

For more information, see Macros.

New parameter types

In DataStage jobs you can use two new parameter types:

Boolean

Use the Boolean parameter type to specify a true or false value.

List

Use the List parameter type to specify a list of values that are available for selection in a job.

For more information about parameters, see Creating and using parameters and parameter sets.

Add and edit column metadata for data definitions

In data definitions, you can now add and edit metadata properties at the column level. For example, you can set properties such as field level, delimiter, quotation mark, and string type.

For more information about data definitions, see Defining data definitions.

Copy and paste subflows

You can copy and paste shared subflows within a DataStage flow or between different DataStage flows in the same project. You can copy and paste a subflow as part of a larger flow or as just the subflow itself.

For more information about subflows, see Subflows.

Oracle, Snowflake, and Teradata connectors can now have multiple input links

Previously the Oracle, Snowflake, and Teradata connectors had only one input link, and you specified the link's properties in the Stage properties.

Now, the connectors can have multiple input links, and each link can have a different property. Therefore, each link can have an individual action, such as read, write, or append. You can view the properties by switching the links in the Input tab.

The 4.5.2 release of DataStage includes the following features and updates:

Send reports by using before/after-job subroutines

You can now send full job reports by using the Send mail function of before/after-job subroutines.

Support for migrating Db2 server-type data connection objects from traditional DataStage

Traditional DataStage supports data connection objects of the type Db2 server. When you migrate these data connection objects to modern DataStage, they are automatically converted to Db2 connector objects so that you can still use them in your DataStage flows and jobs.

New Custom stage

You can now use the Custom stage in your DataStage flows. Use the Custom stage to process your data according to your own specifications.

For details on the Custom stage, see Defining custom stages in DataStage.

Use function libraries in the Transformer stage

You can now define your own functions for use in the Transformer stage's expression editor by creating a function library asset from an SO file. For details on function libraries, see Function libraries in DataStage.

Use new functions in the Transformer stage

You can now use the ConvertDatum and NextValidDate functions in the Transformer stage as part of your DataStage flows.

For the full list of available functions, see Parallel transform functions.

Migrate nested sequence jobs to Watson Pipelines

A traditional DataStage sequence job can contain a "Job Activity" type that can invoke another sequence job, making the second sequence job a nested sequence job.

You can now migrate these nested sequence jobs into modern DataStage. When migrated, a sequence job in the ISX import file is converted to a Watson pipeline.

Connect to a new data source

You can now include data from Apache HBase in your DataStage flows.

For the full list of DataStage connectors, see DataStage connectors.

Kerberos authentication supported for Apache Hive connections

You can now use the Kerberos authentication protocol to connect to an Apache Hive data source when you create the connection from the DataStage service. For information, see Apache Hive connection.

The Db2 (optimized) connector can now have multiple input links, each with an individual action

Previously the Db2 (optimized) connector had only one input link, and you specified the link's properties in the Stage properties. Now the connector can have multiple input links, and each link can have a different property. This enhancement means that each link can have an individual action, such as read, write, append, etc. You can view the properties by switching the links in the Input tab.