File buffer function in the Data Streamer

Edit online

The Data Streamer can buffer data when its TCP/IP connections with the analytics platforms are not stable. When the Data Streamer is stopped, any data that is buffered and not sent to the analytics platforms is written to one or more files in the UNIX System Services file system. When the Data Streamer is restarted, the UNIX System Services files are read by the new instance of the Data Streamer, and the buffered data is sent to the analytics platforms.

The Data Streamer maximum heap size

To handle large volumes of data, the Data Streamer uses 64-bit virtual storage. You can set the limit of the storage amount when you configure the Data Streamer by specifying the value for the parameter MAXIMUM_HEAP. This maximum heap size value is found under the STDENV DD statement in the JCL procedure that you use to start the Data Streamer address space. See the following example:

//STDENV   DD *
JAVA_HOME=/Java/J8.0_64
CDP_HOME=/u/dhods/SYSG/zcdp/V5R1M0/DS/data
TZ=EST5EDT
DEFAULT_HEAP=4g
MAXIMUM_HEAP=4g

The value for the parameter MAXIMUM_HEAP must be no less than the value for the parameter DEFAULT_HEAP. In this example, MAXIMUM_HEAP=4g means that the limit of virtual storage for the Data Streamer address space is 4 GB.

If your mainframe environment has sufficient real storage to support large usage of virtual storage, you can set the parameter MAXIMUM_HEAP to a value higher than 4g. Otherwise, a large MAXIMUM_HEAP value might cause high amounts of paging activities between virtual, real, and auxiliary storage.

The data buffer and UNIX System Services file system size

In general, if the TCP/IP connection between the Data Streamer and the analytics platform is stable, set the maximum heap size to 4 GB. However, if the connection is unstable or lost, the Data Streamer might require a larger amount of virtual storage to buffer data.

If the Data Streamer address space is stopped, any data that is buffered and not sent to the analytics platforms is written to one or more files in the UNIX System Services file system on your z/OS® LPAR. The amount of data can be as large as the maximum heap size that is specified for the Data Streamer address space. Therefore, you must ensure that the UNIX System Services file system has at least the same available storage as the maximum heap size of the Data Streamer address space. For example, if the value of MAXIMUM_HEAP is 4g, the UNIX System Services file system must have at least 4 GB available space to hold the buffered data from the Data Streamer.