Data compression by using the zlibNX library
zlibNX library is an enhanced version of the
compression library that supports hardware-accelerated data compression and decompression by using
co-processors called Nest accelerators (NX) on IBM® POWER9™ processor-based servers. The
zlibNX library is available in IBM AIX® 7.2 with Technology Level 4 Expansion Pack,
and later. Starting with IBM AIX® 7.2 Technology Level 5 Service Pack 2, the
zlibNX installation package is available on the AIX base media in addition to the Expansion
zlibNX library conforms to the following Request For Comment (RFC)
- RFC 1950 (zlib compressed data format specification)
- RFC 1951 (DEFLATE compressed data format specification)
- RFC 1952 (gzip file format specification)
The RFC standards ensure compatibility among different
implementations. Data that is compressed by using the
zlibNX library can be
decompressed by using the standard
zlib library. Similarly, data that is compressed
by using a standard
zlib library can be decompressed by using the
Enabling the applications to use the zlibNX interface
zlibNXinterface based on how the application links to the
libz.alibraries as follows:
- Applications that do not link to the
libz.so.1libraries must be recompiled or relinked to the
zlibNXarchive file and the
zlibNXheader files that are specified in the /usr/opt/zlibNX/include/ path.
- Applications that statically link to the
libz.alibrary can use the
/usr/opt/zlibNX/static/lib/libz.apath to access the
Applications that dynamically link to the
libz.alibrary can use the /usr/opt/zlibNX/lib/libz.a path. Dynamically linked applications can link to the hardware accelerated
zliblibrary by using one of the following ways:
- Set the LDR_PRELOAD or LDR_PRELOAD64 variable to load the
zliblibrary before you load the standard
zliblibrary as shown in the following methods:
# LDR_PRELOAD="/usr/opt/zlibNX/lib/libz.a(libz.so.1)" <32-bit application> # LDR_PRELOAD64="/usr/opt/zlibNX/lib/libz.a(libz.so.1)" <64-bit application>
- Set the LD_LIBRARY_PATH variable as shown in the following
# LD_LIBRARY_PATH=/usr/opt/zlibNX/lib:$LD_LIBRARY_PATH <application>
- Set the LIBPATH variable as shown in the following
A disadvantage of setting the LD_LIBRARY_PATH variable is that the loader attempts to find all the required libraries in the specified directory first.
# LIBPATH=/usr/opt/zlibNX/lib:$LIBPATH <application>
dlopensubroutine, you must update the application with the path to the hardware-accelerated
- Set the LDR_PRELOAD or LDR_PRELOAD64 variable to load the hardware accelerated
Optimizing the zlibNX library
Most of the existing
zlib applications do not use large I/O memory buffer for
zlibNX performance. You can use the following guidelines to increase the
size of the I/O buffer that is used by the applications:
- For hardware-accelerated data decompression, the output memory buffer can be N times larger than
the input memory buffer, where N indicates the expected compression ratio. The value of N depends on
the randomness of the data. To get best compression performance, you must provide sufficient
compressed input data to the
- For hardware-accelerated data compression, you can choose I/O memory buffers that are of equal size.
Environment variables of the zlibNX library
zlibNXlibrary uses the following environment variables to collect error and debug information. The default values for the following environment variables can be overridden.
- Enables different levels of low-level debug information that is printed to the log file. A value of 0 does not write the debug information into the log file. A value of 3 writes the verbose debug information into the log file. The default value is 0.
- Prints the debug information into the specified log file. When the ZLIB_LOGFILE environment
variable is not set, debug information is printed to the
- Enables or disables hardware-accelerated data decompression by using the
deflatefunction in the
zlibNXlibrary. A value of 0 disables the hardware-accelerated data compression. A value of 1 enables the hardware-accelerated data compression. The default value is 1.
- Enables or disables hardware-accelerated data decompression by using the
inflatefunction in the
zlibNXlibrary. A value of 0 disables the hardware-accelerated data decompression. A value of 1 enables the hardware-accelerated data decompression. The default value is 1.
Supported zlib functions
zlibfunctions that are used by the NX GZIP accelerator. Some
zlibfunctions cannot be performed by using the NX accelerator. In those cases, compression or decompression falls back to software-based operations.
- Returns the version of the
zlibNXlibrary. For example:
- deflateInit(z_streamp strm, int level)
- Initializes the compression operation. The value specified by the user for the
level parameter is ignored and instead the value of the
Z_DEFAULT_COMPRESSIONlevel is used for the compression operation.
- deflate(z_streamp strm, int flush)
Compresses data. The data compression is stopped when the input memory buffer that is used by the compression operation becomes empty or when the output memory buffer that is used by the compression operation becomes full.
Introduces output latency when the input data is buffered to meet the minimum threshold for compressing data by using NX accelerator. If the size of the input memory buffer is not sufficient, traditional software-based compression is used.
You can specify one of the following values for the flush parameter:
- Determines the amount of data, which is accumulated in the input memory buffer, used by the compression operation before the compressed output is generated.
- Sends all pending output of the compression operation to the output memory buffer and the output ends on a byte boundary. Byte boundary is end of a byte in the memory.
Compresses available input data and sends the pending output of the compression operation. The output does not end on byte boundary, that is, an empty block of memory is appended to the output data.
The functionality of the
Z_PARTIAL_FLUSHfunction is similar to the
- Indicates that compression of a data block is completed. If the
Z_BLOCKvalue is passed, the
zlibNXAPI falls back to the standard
- Sends all pending output of the compression operation to the output memory buffer and output ends on a byte boundary. Resets the compression state.
- Transfers the pending input data to the deflate function and flushes the pending output from the
deflate function. The
deflatefunction returns the
Z_STREAM_ENDvalue if the output memory is sufficient. If the
Z_BUF_ERRORvalue, the deflate function must be called again with the
Z_FINISHvalue for the flush parameter and with more output space but with no more input data, until the deflate function returns with the Z_STREAM_END value or an error.
- deflateEnd(z_streamp strm)
- Frees the allocation state information for the compression operation.
deflateInit2(z_streamp strm,int level, int method, int windowBits, int memLevel, int strategy)Initializes the compression operation. You can specify the following parameters with this function:
- You can specify one of the following values for the level parameter:
- Compression falls back to software-based compression.
- Currently, this level is considered as the
Z_DEFAULT_COMPRESSIONlevel, and it is the default level.
- Levels 1 - 9 are treated as level 6, that is,
Z_DEFAULT_COMPRESSIONlevel, which is the default level.
- You can specify one of the following values for the method parameter:
- If the method parameter has a value other than
Z_DEFLATED, compression falls back to the software-based decompression.
- Indicates the size of the internal history buffer. Accepts value of the base 2 logarithm. The
size of the history buffer (window size) can be in the range 256 bytes - 32 KB. The
windowBits parameter impacts memory that is allocated for the internal input
memory buffer. You can specify values in the following range for the windowBits parameter:
- 8 - 15
- Indicates the
- -8 to -15
- Indicates the raw deflate format. The raw deflate format means a compressed data format.
- 24 - 31
- Indicates the GZIP format.
deflateInit2function accepts the requested format, but always uses a 32 KB window or history buffer.
- Indicates the amount of memory that must be allocated for the internal compression state.
Internal compression state is a data structure that is used in the software library to track
compression related information. The value can be in the range 1 - 9. The default value is 8. This
value is currently ignored. The
zlibNXAPI always allocates a large internal compression state (approximately 1 MB).
Indicates the algorithm that must be used for the compression operation. You can specify one of the following values for the strategy parameter:
- The dynamic Huffman algorithm with string-matching algorithm is used.
- The dynamic Huffman algorithm is not used.
- Only Huffman algorithm without string-matching is used. The compression operation falls back to software-based compression.
- Data produced by a filter or a predictor is the input to the deflate function. The compression operation falls back to software-based compression.
- The Huffman encoding algorithm with limited string matching distances is used. The compression operation falls back to software-based compression.
- inflateInit(z_streamp strm)
- Initializes the decompression operation.
- inflate(z_streamp, int flush)
- Decompresses data until the input memory buffer becomes empty or until the output memory buffer
becomes full. Hardware-based acceleration depends on the sufficiently large input memory buffer that
is used by the compression operation. Otherwise software-based compression operation is used.
Default threshold value for the input memory buffer is 4 KB.The flush parameter can take one of the following values:
- Decompress the data until end of the input or the end of the output memory buffer is reached.
- Sends output of the decompression operation to the output memory buffer until the maximum capacity of the output memory buffer is reached.
- Attempts to decompress all data in a single iteration of the decompression operation.
- Stops the decompression operation if the next deflate block boundary is reached. The block boundary refers to the location where a deflate block ends.
InflateEndfunction is supported by the
Initializes the compression dictionary with values specified in byte sequence (dictionary parameter). The compression dictionary must consist of strings that might be encountered later in the data that must be compressed.
deflateSetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)
Returns sliding dictionary (history buffer) that is maintained by the
deflateGetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)
deflatefunction. Sliding dictionary is a memory buffer containing uncompressed data.
Copies a stream and duplicates the internal compression state. The destination stream might not be able to access the accelerator.
deflateCopy(z_streamp dest, z_streamp source)
Resets the state of a stream without freeing and reallocating internal compression state.
Supports the deflate function in the
deflateParams(z_streamp strm, int level, int strategy)
zlibNXlibrary based on input parameters. Compression level does not change.
Refines the internal compression parameters of the
deflateTune(z_streamp strm, int good_length, int max_lazy,int nice_length, int max_chain)
deflatefunction. The compression operation falls back to software-based compression.
- deflateBound(z_streamp strm, uLong sourceLen)
- Returns the maximum size of the compressed data after the
deflateoperation is performed on the sourceLen bytes that is passed to the
deflatePending(z_streamp strm, unsigned *pending, int *bits)
Returns the output that is generated but not yet displayed in the output of the compression operation.
The compression operation falls back to software-based compression.
deflatePrime(z_streamp strm, int bits, int value)
- deflateSetHeader(z_streamp strm, gz_headerp head)
- Provides the GZIP header information of the specified GZIP stream.
Similar to the
inflateInit2(z_streamp strm, int windowBits)
inflateinitfunction, but accepts an additional parameter, windowBits.
- Accepts value of base 2 logarithm. The size of the history buffer (window size) can be in the
range 256 bytes - 32 KB. The
deflateInit2function accepts the requested format, but always uses a 32 KB window or history buffer. You can specify values in the following range for the windowBits parameter:
- 8 - 15
- Indicates the
- 24 - 31
- Indicates the GZIP format
- 40 - 47
- Indicates automatic header detection.
- -8 to -15
- Indicates raw deflate format.
inflateInit2function accepts the requested format, but always uses a 32KB window or history buffer.
Initializes the decompression dictionary from the specified decompressed byte sequence.
inflateSetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)
inflateGetDictionary(z_streamp strm, Bytef *dictionary, uInt *dictLength)
- Returns the decompression dictionary that is set in the dictionary parameter.
Skips compressed data during the inflate operation until a possible full flush point is reached. If a full flush point is not reached, the inflate operation skips all the available input data. A full flush point refers to a location in compressed data that was generated when the
deflate()function was called with the flush parameter set to the
Sets the destination stream as a copy of the source stream.
inflateCopy(z_streamp dest, z_streamp source)
Resets the compression state but does not free the compression state.
inflateReset2(z_streamp strm, int windowBits)
inflateReset2function is similar to the
inflateResetfunction, but it also allows changing the
wrapand window size values. The
windowBitsparameter is used similarly as it is used in the
inflateInit2function. Size of the history buffer (window size) is static at 32 KB, but the format that is indicated by the
windowBitsparameter is used for decompression.
Inserts bits in the inflate input stream. This function is used to start the inflate operation at a bit position within a byte. The compression operation falls back to software-based decompression.
inflatePrime(z_streamp strm, int bits, int value)
Marks location to randomly access data from input stream, which is specified as bit positions in the data stream. Returns two 16-bit values that are upper and lower halves of 32-bit input data.
Stores the GZIP header information in the specified header structure.
inflateGetHeader(z_streamp strm, gz_headerp head)
Initializes the internal stream for the decompression operation by calling the
inflateBackfunction. The decompression operation falls back to software-based decompression. This function is supported only at the start of a stream.
Performs a raw inflate operation by using a call-back interface for input and output data. The decompression operation falls back to software-based decompression. This function must be called only after calling the
inflateBack(z_streamp strm,in_func in, void FAR *in_desc,out_func out,void FAR *out_desc)
Free all memory that is allocated by the
inflateBackInitfunction. The decompression operation falls back to software-based decompression.