Overview of IBM DataStage Flow Designer

1. Drag a database connector from the palette to the canvas, which opens the Table Definition Asset Browser. This browser shows a schema representation of the connection. You use the Table Definition Asset Browser to configure the connection.
2. Drag the ‘Connections’ connector from the palette to the canvas, which opens the Live Connection Asset Browser. From the browser, you pick an existing connection/schema/table/column. When you choose a table from the Live Connection Asset Browser, you also have the option of selecting an available view that was created by using the Data Virtualization service. Because the connection exists already, the connection details, such as URL, user ID, password, and table, are added to the connector automatically.

Find your connector or stage quickly by using the find icon in the palette.

To delete an object from the canvas, select the object on the canvas that you want to delete, then click the Delete icon on the toolbar. Or select the object and press DELETE on your keyboard if you are using Windows. Press FN + DELETE if you are using a Mac.

To create a container, drag the container icon from the palette to the canvas, then double-click the container icon to open it. An input and output stage are automatically provided in the container. Add additional stages and links in between the input and output stages, then rename the input and output links to match the names of the links going into and coming out of the container itself.

Job parameters allow you to specify properties and values for job parameters without having to define them first. At job runtime, the parameters can either use a defined default or a new value. This allows jobs to be reusable and flexible. For example, you might define a job for a particular test database and when the job is thoroughly tested and ready for production, key connector files such as database name, table name, user, and password can be substituted with job parameters and deployed to a production environment.

You can create, edit, and delete Job parameters of type encrypted, date, integer, float, pathname, date and time, and environment variable, as well as configuration files for parallel and sequence jobs.

For Spark jobs, all connector properties for a connection, such as host, port, database, user name, table name, row limit, and byte limit, are supported as a string type Job parameter. SSH parameter substitution is not supported for Spark jobs. The certificate must be explicitly identified in the property for specific connections such as the HDFS connector.

After you define Job parameters you are prompted to either accept the parameter defaults or replace them when the job is run.

You can also create, edit, and delete List type parameters. List type parameters can contain multiple string values, giving you more flexibility and efficiency in your jobs.

Use parameter sets to define job parameters that you are likely to reuse in different jobs, such as connection details for a particular database. Then, when you need this set of parameters in a job design, you can insert them into the job properties from the parameter set. You can also define different sets of values for each parameter set.

The Smart Palette feature arranges the connectors and stages in the most used fashion. This arrangement keeps your favorite connectors and stages on top.

The Suggested Stages feature suggests stages when you click a stage on the canvas that has no outputs. The suggested stages are highlighted in the palette and displayed on the canvas with dotted lines. You can select one of the suggestions by either dragging the suggested stage from the palette or clicking the suggested stage on the canvas. When you add a stage by either method, the added stage is automatically linked to the stage that you clicked that had no outputs.

You can configure partitioning for your data to help ensure an even load across your processors. For more information on partitioning, see Partitioning.

Preference settings are automatically saved for the next session.

If you are an administrator, you set up access to Git by clicking your profile icon, then clicking Setup > Server > Git.

If you are a regular user, you manage your Git settings by clicking your profile icon, then clicking Setup > Git user. If you do not have a Git repository ID, you are given the option to clone the Git repository.

Column name

The name of the column.

Key

Indicates whether the column is part of the primary key.

SQL type

The SQL data type.

Extended

This column gives you further control over data types used in parallel jobs when National Language Support is enabled. The available values depend on the base data type:

• For Char, VarChar, and LongVarChar, you can select Unicode to specify that these columns require mapping (by default, each character is assumed to represent an ASCII character that does not need mapping).
• For Time, you can select microseconds to indicate that the field contains microseconds. (By default, a Time type comprises hours, minutes, and seconds)
• For Timestamp, you can select microseconds to indicate that the field contains microseconds. (By default, the time part of a Timestamp type comprises hours, minutes, and seconds)
• For integer types, you can select unsigned to specify that the underlying data type is a unit (unsigned integer)

Length

The data precision. This is the length for Char data and the maximum length for VarChar data.

Scale

The data scale factor. (For Sequential File stages the scale should not exceed 9.)

Nullable

Specifies whether the column can contain null values. This field is set to indicate whether the column is subject to a NOT NULL constraint. The field does not itself enforce a NOT NULL constraint.