Operational Decision Manager (ODM) stage field propagation strategies in DataStage®

The Operational Decision Manager (ODM) stage supports passing through unmatched fields from the input links to the output links. Unmatched fields are input link fields that are not used by the connector to invoke the ruleset. When you enable the propagation of unmatched fields, then the unmatched fields are propagated to the output links associated with the input links.

The connector automatically establishes the association between each input link that has the unmatched fields and the output links to which those fields must be propagated. For every unmatched field on an input link and for every output link that is associated with that input link, the connector checks if the output link contains the field with the same name as the unmatched field.

If it does and if the output field is not mapped to any field in the XOM, then the field values of the input column are propagated and set as the field values of the output column.
If it does not, the connector checks if the Runtime column propagationcheckbox is selected for that output link. If it is selected, the connector adds the input field definition to that output link schema and propagates the field values as the job runs. If the checkbox is not selected, the propagation of the input field definition and values does not take place.

You can use one of the following strategies to propagate unmatched fields:

Matching based on the order of records

When this strategy is selected, the connector propagates unmatched fields from the first input link to the first output link, from the second input link to the second output link, and so on. If there are more input links than the output links, then the field propagation is not performed for the input links for which there are no corresponding output links. An input link can be associated with only one output link. Similarly, an output link can be associated with only one input link.

When this strategy is used, the number of records on the input link that were used to invoke the ruleset must match the number of records that are produced on the associated output link after the ruleset runs. If the numbers do not match, the connector logs an error and the job fails.

Matching based on the propagation key column value

When this strategy is used, the connector propagates unmatched fields from an input link to an output link with the same propagation key column. The propagation key column is specified for each input link separately, through the Propagation key column link property.

The connector checks each input link to verify whether there is a column that is marked as the propagation key column. If input links with column marked as propagation key column are found, and if the links have fields that are not used for ruleset execution, those fields are eligible for field propagation.

A propagation key column cannot be a column that is eligible for field propagation at the same time. If this scenario occurs, the job fails with an error. The connector inspects the input links and for each input link that has propagation key column as well as the columns eligible for propagation, the connector performs the following steps:

It inspects all the output links that are not already associated with any of the previous input links.
The output link that includes column with the same name and the data type as the propagation key column of the input link becomes associated with that input link.

With this strategy an input link can be associated with multiple output links, but an output link can be associated with only one input link.

When the connector reads records on an input link with the propagation key column enabled, it checks the propagation key column value of the record. If the value matches the value of another record that is retrieved on the same link for the current ruleset execution, the job fails with an error.

For each record that is sent to an output link, the record field is populated from the value of the respective OUT or IN_OUT parameter of the ruleset. The connector then checks whether the output link is associated with an input link for field propagation. If it is associated, the connector searches for the input record that it retrieved from the input link for the current ruleset execution and that has the same propagation key column value as the output record. If such a record exists, the connector copies the values of the propagated fields from the input record to the output record. If such an input record does not exist, the propagated fields in the output record remain unset.

Consider an example of a job where the ODM stage has two input links and two output links.

The links contain the columns as follows:

Input links

Link 1: (id, balance)

Link 2: (id, name, address)

Output links

Link 1: (name, cust_type, address)

Link 2: (name, outstanding, address)

If field propagation is enabled using the propagation key column strategy and the column name is specified as the propagation key column for Link 2, the connector detects that the name column is also defined on the two output links. If the address column is not being mapped to the ruleset in both the input as well as the output link, then the data of the address column from input Link 2 is sent to the address column of both the output links. The mapping is done based on the name field. For example, if input Link 2 has the following data:

1, "John", "123, Las Vegas Dr, Las Vegas, NV"

2, "Paul", "abc Drive, New York"

Records on the output links show up as follows:

Link 1

"John", "Gold", "123, Las Vegas Dr, Las Vegas, NV"

"Paul", "Silver", "abc Drive, New York"

Link 2

"John", "244.33", "123, Las Vegas Dr, Las Vegas, NV"

"Paul", "435.6445", "abc Drive, New York"

"Manish", 4363.453", ""

The data from the input address column is mapped to the output address column by using the value of the name column. The name field value in the third record on output Link 2 does not match the name field value in any of the records on input Link 2, so the address field value in the output record remains unset.

Matching based on the Java Object identifiers

When this strategy is used, the connector propagates unmatched fields from an input record to the output record that corresponds to the same Java™ object. For each input link, the connector checks the Java type of the ruleset parameter that is specified for the link. If this type is not one of the basic Java types that are supported by the connector, the connector makes note of any columns on the link that are eligible for field propagation, meaning they are not used for producing the ruleset parameter values.

For each input link that has some columns (fields) eligible for propagation, the connector internally establishes association between that input link and the output links as follows:

It inspects all the output links that are not already associated with any of the previous input links.
The output links whose ruleset parameters are of the compatible Java types with the one specified for the input link become associated with the input link. Two Java types are compatible in this context if they are identical or if one of them is derived from the other. The Java array types are treated as the nonarray types when compatibility is determined. For example if one Java type is ClassA and the other Java type is ClassB[], the two Java types are compatible if ClassA and ClassB are either identical or if one is derived from the other.

With this field propagation strategy in place, one input link can be associated with multiple output links, but one output link can be associated with only one input link.

For each record that the connector sends to an output link, it first populates the record fields from the value of the respective OUT or IN_OUT parameter of the ruleset and then checks whether that output link is associated with an input link for the field propagation purposes. If it is associated, then the connector looks for the record that is retrieved from the input link for the current ruleset execution and that corresponds to the same Java object as the output record. If such a record exists, the connector copies the values for the propagated fields from the input record to the output record. If such a record does not exist, the propagated fields in the output record remain unset.