External Filter stage

The External Filter stage is a processing stage that allows you to specify a UNIX command that acts as a filter on the data you are processing.

The External Filter stage is a processing stage. It can have a single input link and a single output link.

The External Filter stage allows you to specify a UNIX command that acts as a filter on the data you are processing. An example would be to use the stage to grep a data set for a certain string, or pattern, and discard records which did not contain a match. This technique can be a quick and efficient way of filtering data.

Shows a job where an External filter stage takes input from a data set and outputs it to another data set

Whitespace is stripped from the start and end of the data before the command is executed. To avoid this behavior, use an explicitly wrapped command that sets format options on the schema.

The stage editor has three pages:

Stage Page. Use this page to specify general information about the stage.
Input Page. Use this page to specify details about the input link carrying the data to be filtered.
Output Page. Use this page to specify details about the filtered data being output from the stage.