External Filter stage (DataStage)
The External Filter stage allows you to specify a UNIX command that acts as a filter on the data you are processing.
The External Filter stage is a processing stage. It can have a single input link and a single output link. For example, you can use the stage to grep a data set for a certain string, or pattern, and discard records that do not contain a match. This technique can be used to quickly filter data.
Whitespace is stripped from the start and end of the data before the command is executed. To avoid this behavior, use an explicitly wrapped command that sets format options on the schema.
The stage editor has three tabs:
- Stage . Use this tab to specify general information about the stage.
- Input. Use this tab to specify details about the input link carrying the data to be filtered.
- Output. Use this tab to specify details about the filtered data being output from the stage.
Input tab
The Columns section specifies the column definitions of incoming data. The Advanced section allows you to change the default buffering settings for the input link.
Output tab
The Output tab allows you to specify details about data output from the External Filter stage. The stage can only have one output link.
Click Edit in the Columns section to specify the column definitions of the data. The Advanced section allows you to change the default buffering settings for the output link.