Buffering

Buffering is intended to slow down input to match the consumption rate of the output. When the downstream operator reads slowly, or not at all, for a length of time, upstream operators begin to slow down. This slowing down causes a noticeable performance loss if the buffer's optimal behavior is something other than rate matching.

By default, each link has a 3 MB in-memory buffer. Once that buffer reaches half full, the operator begins to push back on the upstream operator's rate. Once the 3 MB buffer is filled, data is written to disk in 1 MB chunks.

In most cases, the easiest way to tune buffering is to eliminate the pushback and allow it to buffer the data to disk as necessary. Setting APT_BUFFER_FREE_RUN=N or setting Buffer Free Run in the Output page Advanced tab on a particular stage will do this. A buffer will read N * max_memory (3 MB by default) bytes before beginning to push back on the upstream. If there is enough disk space to buffer large amounts of data, this will usually fix any egregious slowdown issues cause by the buffer operator.

If there is a significant amount of memory available on the machine, increasing the maximum in-memory buffer size is likely to be very useful if buffering is causing any disk I/O. Setting the APT_BUFFER_MAXIMUM_MEMORY environment variable or Maximum memory buffer size on the Output page Advanced tab on a particular stage will do this. It defaults to 3145728 (3 MB).

For systems where small to medium bursts of I/O are not desirable, the 1 MB write to disk size chunk size might be too small. The environment variable APT_BUFFER_DISK_WRITE_INCREMENT or Disk write increment on the Output page Advanced tab on a particular stage controls this and defaults to 1048576 (1 MB). This setting might not exceed max_memory * 2/3.

Finally, in a situation where a large, fixed buffer is needed within the flow, setting Queue upper bound on the Output page Advanced tab (no environment variable exists) can be set equal to max_memory to force a buffer of exactly max_memory bytes. Such a buffer will block an upstream operator (until data is read by the downstream operator) once its buffer has been filled, so this setting should be used with extreme caution. This setting is rarely, if ever, necessary to achieve good performance, but might be useful in an attempt to squeeze every last byte of performance out of the system where it is desirable to eliminate buffering to disk entirely. No environment variable is available for this flag, and therefore this can only be set at the individual stage level.