End-of-wave batch processing

Using the end-of-wave functionality, you can perform a minimal set of transaction-like processes.

The functionality allows you to put records into distinct groups, called waves, which flow through for batch-processing by operators that perform aggregations, sorts, transformations, and so on.

Some typical transactional functionality is not currently supported, such as rollbacks when end-of-wave failures occur, or the ability to restart reading from a particular persistent data set entry. Also, end-of-wave processing is currently not applicable to work flows that require parallel-to-sequential partition collection.

When an operator encounters an end-of-wave marker, getRecord() returns APT_InputCursor::eAtEndOfWave , and each set of records is appended as a batch to the output data set or data sets or database table or tables.

Waves provide functionality that is like repeatedly invoking osh, possibly with different arguments to read different sets of input data. However, waves avoid the overhead of restarting a job. For regular non-transaction-like data processing, the entire input data is treated as a single wave but there is no end-of-wave marker seen before the end of file. For transaction-like data processing with a single wave, the wave is treated as a file but with an end-of-wave marker seen before the end of file.