Creating a Complex Pipeline

This section describes the way to create a complex pipeline using stages with multiple inputs and outputs. When stages are adjacent to each other in a pipeline, the output stream of a stage is connected to the input stream of the stage that follows.

Use a label to connect the streams of stages that are not adjacent. A label is 1–8 alphanumeric characters followed by a colon. For example, the B in the following example is a label:

...|B: LOCATE /X/|...

To use multiple streams, first include the label on the stage that results in multiple output streams. This first label defines or declares the label. Then, place a matching label in the pipeline specification as if it were a stage. The stages following the label will act on the data passed as the primary output of the stage defining the label. The label acts as a connector and provides the input stream to the subsequent pipeline stages, for example:

PIPE (END %)
     < NAMES
     | A: LOCATE /BOB/
     | CHANGE //HERE IS A NAME CONTAINING 'BOB' ==>/
     | CONSOLE
     %A:
     | CONSOLE

The < (From Disk) stage reads data from a file called NAMES, containing three names. The three names are:

BOB SMITH
FRED FORD
MARY BOBBIT

The selected data is written to the console using the CONSOLE stage. All records containing the string BOB will be prefixed with the string HERE IS A NAME CONTAINING 'BOB' ==>.

In this example, the string BOB is located in the input data. Label A: was defined on the LOCATE stage; data that does not contain the string BOB will be passed as an input stream to the stage following the stage labeled A:. The end of the simple pipeline and the beginning of the second is indicated by the end character, which is defined as a % symbol in (END %).

This complex pipeline is logically made up of the following parts:

The definition of the end character that is to be used to separate the different simple pipelines.
```
PIPE (END %)
```
The first simple pipeline. The LOCATE stage, which generates multiple output streams, is labeled with an A: indicating that data not selected by the LOCATE stage will be passed to the connector A: later in the pipeline specification.
```
     < NAMES
     | A: LOCATE /BOB/
     | CHANGE //HERE IS A NAME CONTAINING 'BOB' ==>/
     | CONSOLE
```
The end character indicates the end of the first simple pipeline and the beginning of the second simple pipeline.
```
     %
```
The next occurrence of label A: is as a connector that connects the secondary output of LOCATE /BOB/ as an input stream to CONSOLE in the second simple pipeline. This A: is a connector and not a label definition because this A: is not included in a stage.
```
     A:
```
The second simple pipeline that will handle the data not selected by LOCATE /BOB/.
```
     | CONSOLE
```

The resulting output of this complex pipeline is shown in Figure 1.

Figure 1. Complex Pipeline Example Output

 NCCF                      N E T V I E W    CNM19 PETE     03/26/10  13:10:00
 * CNM19    PIPE (END %) < NAMES | A: LOCATE /BOB/ | CHANGE //HERE IS
            A NAME CONTAINING 'BOB' ==>/| CONSOLE % A: | CONSOLE

 | CNM19    HERE IS A NAME CONTAINING 'BOB' ==>BOB SMITH
 | CNM19    FRED FORD
 | CNM19    HERE IS A NAME CONTAINING 'BOB' ==>MARY BOBBIT

Note:

You can have as many simple pipelines within a complex pipeline specification as you need. Each stage with multiple outputs can pass data to different connectors. Or, multiple stages can pass data to a single connector.
Each pipeline stream acts as independently as possible. For example, in Figure 1, the record FRED FORD is processed in the second simple pipeline, A:, while the first simple pipeline is still processing the records selected by LOCATE /BOB/.
If a connector immediately follows an end character, then it defines an output stream to the stage where the label is defined. If an end character, or the end of the pipeline specification, immediately follows a connector, the connector defines input to the stage where the label was defined. Otherwise, the connector defines both an input and output stream to the stage where the label was defined.