Buffering environment variables in DataStage

These environment variable are all concerned with the buffering DataStage® performs on stage links to avoid deadlock situations.

APT_BUFFER_FREE_RUN environment variable in DataStage

APT_BUFFER_FREE_RUN specifies how much of the available in-memory buffer to consume before the buffer resists.

This variable is expressed as a decimal that represents the percentage of the maximum memory buffer size (for example, 0.5 is 50%). When the amount of data in the buffer is less than this value, new data is accepted automatically. When the data exceeds it, the buffer first tries to write some of the data it contains before the buffer accepts more.

The default value is 50% of the maximum memory buffer size. You can set it to greater than 100%, in which case the buffer continues to store data up to the indicated multiple of the maximum memory buffer size before the buffer writes to disk.

APT_BUFFER_MAXIMUM_MEMORY environment variable in DataStage

APT_BUFFER_MAXIMUM_MEMORY sets the default value of maximum memory buffer size.

The APT_BUFFER_MAXIMUM_MEMORY variable specifies the maximum amount of virtual memory, in bytes, used per buffer. The default value is 3145728 (3 MB).

APT_BUFFER_MAXIMUM_TIMEOUT environment variable in DataStage

APT_BUFFER_MAXIMUM_TIMEOUT specifies the maximum wait, in seconds, before a retry.

DataStage buffering is self-tuning, which can theoretically lead to long delays between retries. This environment variable specifies the maximum wait, in seconds, before a retry, and is, by default, set to 1.

APT_BUFFER_DISK_WRITE_INCREMENT environment variable in DataStage

APT_BUFFER_DISK_WRITE_INCREMENT sets the size, in bytes, of blocks of data that are moved to or from disk by the buffering operator.

The default is 1048576 (1 MB). Adjusting this value trades the amount of disk access against throughput for small amounts of data. Increasing the block size reduces disk access, but might decrease performance when data is being read or written in smaller units. Decreasing the block size increases throughput, but might increase the amount of disk access.

APT_BUFFERING_POLICY environment variable in DataStage

APT_BUFFERING_POLICY controls the buffering policy for all virtual data sets in all steps.

This variable has the following settings:

  • AUTOMATIC_BUFFERING (default). Buffer a data set only if necessary to prevent a data flow deadlock.
  • FORCE_BUFFERING. Unconditionally buffer all virtual data sets. This setting can slow down processing considerably.
  • NO_BUFFERING. Do not buffer data sets. This setting can cause data flow deadlock if used inappropriately.

APT_CONSISTENT_BUFFERIO_SIZE environment variable in DataStage

APT_CONSISTENT_BUFFERIO_SIZE specifies the size in bytes of read and write that buffers use for accessing sequential files, and other similar files such as named pipe access to wrapped operators.

If this variable in not set, the appropriate buffer size is determined on a case-by-case basis, depending on the record size or how the buffer class is being used.

If this variable is set, then set size in used for the buffer size.

This variable has a Number type.