Data compression by using the zlibNX library

Edit online

The zlibNX library is an enhanced version of the zlib compression library that supports hardware-accelerated data compression and decompression by using co-processors called Nest accelerators (NX) on IBM® POWER9™ processor-based servers. The zlibNX library is available in IBM AIX® 7.2 with Technology Level 4 Expansion Pack, and later. Starting with IBM AIX® 7.2 Technology Level 5 Service Pack 2, the zlibNX installation package is available on the AIX base media in addition to the Expansion Pack.

The zlibNX library conforms to the following Request For Comment (RFC) standards:

RFC 1950 (zlib compressed data format specification)
RFC 1951 (DEFLATE compressed data format specification)
RFC 1952 (gzip file format specification)

The RFC standards ensure compatibility among different zlib library implementations. Data that is compressed by using the zlibNX library can be decompressed by using the standard zlib library. Similarly, data that is compressed by using a standard zlib library can be decompressed by using the zlibNX library.

Enabling the applications to use the zlibNX interface

An application can use the zlibNX interface based on how the application links to the libz.so.1 or libz.a libraries as follows:

Applications that do not link to the libz.a or libz.so.1 libraries must be recompiled or relinked to the zlibNX archive file and the zlibNX header files that are specified in the /usr/opt/zlibNX/include/ path.
Applications that statically link to the libz.a library can use the /usr/opt/zlibNX/static/lib/libz.a path to access the zlibNX interface.
Applications that dynamically link to the libz.a library can use the /usr/opt/zlibNX/lib/libz.a path. Dynamically linked applications can link to the hardware accelerated zlib library by using one of the following ways:
- Set the LDR_PRELOAD or LDR_PRELOAD64 variable to load the hardware accelerated zlib library before you load the standard zlib library as shown in the following methods:
```
# LDR_PRELOAD="/usr/opt/zlibNX/lib/libz.a(libz.so.1)" <32-bit application>
# LDR_PRELOAD64="/usr/opt/zlibNX/lib/libz.a(libz.so.1)" <64-bit application>
```
- Set the LD_LIBRARY_PATH variable as shown in the following sample:
```
# LD_LIBRARY_PATH=/usr/opt/zlibNX/lib:$LD_LIBRARY_PATH <application>
```
- Set the LIBPATH variable as shown in the following sample:
```
# LIBPATH=/usr/opt/zlibNX/lib:$LIBPATH <application>
```
  A disadvantage of setting the LD_LIBRARY_PATH variable is that the loader attempts to find all the required libraries in the specified directory first.
If the applications load the libz.so.1 library dynamically by using the dlopen subroutine, you must update the application with the path to the hardware-accelerated zlib library.

Optimizing the zlibNX library

Most of the existing zlib applications do not use large I/O memory buffer for optimal zlibNX performance. You can use the following guidelines to increase the size of the I/O buffer that is used by the applications:

For hardware-accelerated data decompression, the output memory buffer can be N times larger than the input memory buffer, where N indicates the expected compression ratio. The value of N depends on the randomness of the data. To get best compression performance, you must provide sufficient compressed input data to the inflate function.
For hardware-accelerated data compression, you can choose I/O memory buffers that are of equal size.

Environment variables of the zlibNX library

The zlibNX library uses the following environment variables to collect error and debug information. The default values for the following environment variables can be overridden.

ZLIB_VERBOSE: Enables different levels of low-level debug information that is printed to the log file. A value of 0 does not write the debug information into the log file. A value of 3 writes the verbose debug information into the log file. The default value is 0.
ZLIB_LOGFILE: Prints the debug information into the specified log file. When the ZLIB_LOGFILE environment variable is not set, debug information is printed to the stderr stream.
ZLIB_COMPRESS_ACCEL: Enables or disables hardware-accelerated data decompression by using the deflate function in the zlibNX library. A value of 0 disables the hardware-accelerated data compression. A value of 1 enables the hardware-accelerated data compression. The default value is 1.
ZLIB_DECOMPRESS_ACCEL: Enables or disables hardware-accelerated data decompression by using the inflate function in the zlibNX library. A value of 0 disables the hardware-accelerated data decompression. A value of 1 enables the hardware-accelerated data decompression. The default value is 1.

Supported zlib functions

The following list describes the zlib functions that are used by the NX GZIP accelerator. Some zlib functions cannot be performed by using the NX accelerator. In those cases, compression or decompression falls back to software-based operations.

zlibVersion

Returns the version of the zlibNX library. For example: 1.2.11.1-AIX_NX.

deflateInit(z_streamp strm, int level)

Initializes the compression operation. The value specified by the user for the level parameter is ignored and instead the value of the Z_DEFAULT_COMPRESSION level is used for the compression operation.

deflate(z_streamp strm, int flush)

Compresses data. The data compression is stopped when the input memory buffer that is used by the compression operation becomes empty or when the output memory buffer that is used by the compression operation becomes full.

Introduces output latency when the input data is buffered to meet the minimum threshold for compressing data by using NX accelerator. If the size of the input memory buffer is not sufficient, traditional software-based compression is used.

You can specify one of the following values for the flush parameter:

Z_NO_FLUSH

Determines the amount of data, which is accumulated in the input memory buffer, used by the compression operation before the compressed output is generated.

Z_SYNC_FLUSH

Sends all pending output of the compression operation to the output memory buffer and the output ends on a byte boundary. Byte boundary is end of a byte in the memory.

Z_PARTIAL_FLUSH

Compresses available input data and sends the pending output of the compression operation. The output does not end on byte boundary, that is, an empty block of memory is appended to the output data.

The functionality of the Z_PARTIAL_FLUSH function is similar to the Z_SYNC_FLUSH function.

Z_BLOCK

Indicates that compression of a data block is completed. If the Z_BLOCK value is passed, the zlibNX API falls back to the standard zlib API.

Z_FULL_FLUSH

Sends all pending output of the compression operation to the output memory buffer and output ends on a byte boundary. Resets the compression state.

Z_FINISH

Transfers the pending input data to the deflate function and flushes the pending output from the deflate function. The deflate function returns the Z_STREAM_END value if the output memory is sufficient. If the deflate function returns the Z_OK or Z_BUF_ERROR value, the deflate function must be called again with the Z_FINISH value for the flush parameter and with more output space but with no more input data, until the deflate function returns with the Z_STREAM_END value or an error.

deflateEnd(z_streamp strm)

Frees the allocation state information for the compression operation.

deflateInit2

deflateInit2(z_streamp strm,int level, int method,
int windowBits, int memLevel, int strategy)

Initializes the compression operation. You can specify the following parameters with this function:

level

You can specify one of the following values for the level parameter:

Z_NO_COMPRESSION: Compression falls back to software-based compression.
Z_BEST_SPEED: Currently, this level is considered as the Z_DEFAULT_COMPRESSION level, and it is the default level.
Z_BEST_COMPRESSION: Levels 1 - 9 are treated as level 6, that is, Z_DEFAULT_COMPRESSION level, which is the default level.

method

You can specify one of the following values for the method parameter:

Z_DEFLATED: If the method parameter has a value other than Z_DEFLATED, compression falls back to the software-based decompression.

windowBits

Indicates the size of the internal history buffer. Accepts value of the base 2 logarithm. The size of the history buffer (window size) can be in the range 256 bytes - 32 KB. The windowBits parameter impacts memory that is allocated for the internal input memory buffer. You can specify values in the following range for the windowBits parameter:

8 - 15: Indicates the zlib format.
-8 to -15: Indicates the raw deflate format. The raw deflate format means a compressed data format.
24 - 31: Indicates the GZIP format.

The deflateInit2 function accepts the requested format, but always uses a 32 KB window or history buffer.

MemLevel

Indicates the amount of memory that must be allocated for the internal compression state. Internal compression state is a data structure that is used in the software library to track compression related information. The value can be in the range 1 - 9. The default value is 8. This value is currently ignored. The zlibNX API always allocates a large internal compression state (approximately 1 MB).

strategy

Indicates the algorithm that must be used for the compression operation. You can specify one of the following values for the strategy parameter:

Z_DEFAULT_STRATEGY: The dynamic Huffman algorithm with string-matching algorithm is used.
Z_FIXED: The dynamic Huffman algorithm is not used.
Z_HUFFMANONLY: Only Huffman algorithm without string-matching is used. The compression operation falls back to software-based compression.
Z_FILTERED: Data produced by a filter or a predictor is the input to the deflate function. The compression operation falls back to software-based compression.
Z_RLE: The Huffman encoding algorithm with limited string matching distances is used. The compression operation falls back to software-based compression.

inflateInit(z_streamp strm)

Initializes the decompression operation.

inflate(z_streamp, int flush)

Decompresses data until the input memory buffer becomes empty or until the output memory buffer becomes full. Hardware-based acceleration depends on the sufficiently large input memory buffer that is used by the compression operation. Otherwise software-based compression operation is used. Default threshold value for the input memory buffer is 4 KB.

The flush parameter can take one of the following values:

Z_NO_FLUSH: Decompress the data until end of the input or the end of the output memory buffer is reached.
Z_SYNC_FLUSH: Sends output of the decompression operation to the output memory buffer until the maximum capacity of the output memory buffer is reached.
Z_FINISH: Attempts to decompress all data in a single iteration of the decompression operation.
Z_BLOCK: Stops the decompression operation if the next deflate block boundary is reached. The block boundary refers to the location where a deflate block ends.

InflateEnd

The InflateEnd function is supported by the zlibNX API.

deflateSetDictionary

deflateSetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)

Initializes the compression dictionary with values specified in byte sequence (dictionary parameter). The compression dictionary must consist of strings that might be encountered later in the data that must be compressed.

deflateGetDictionary

deflateGetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)

Returns sliding dictionary (history buffer) that is maintained by the deflate function. Sliding dictionary is a memory buffer containing uncompressed data.

deflateCopy

deflateCopy(z_streamp dest, z_streamp source)

Copies a stream and duplicates the internal compression state. The destination stream might not be able to access the accelerator.

deflateReset

deflateReset(z_streamp strm)

Resets the state of a stream without freeing and reallocating internal compression state.

deflateParams

deflateParams(z_streamp strm, int level, int strategy)

Supports the deflate function in the zlibNX library based on input parameters. Compression level does not change.

deflateTune

deflateTune(z_streamp strm, int good_length, int max_lazy,int nice_length, int max_chain)

Refines the internal compression parameters of the deflate function. The compression operation falls back to software-based compression.

deflateBound(z_streamp strm, uLong sourceLen)

Returns the maximum size of the compressed data after the deflate operation is performed on the sourceLen bytes that is passed to the deflateBound function.

deflatePending

deflatePending(z_streamp strm, unsigned *pending, int *bits)

Returns the output that is generated but not yet displayed in the output of the compression operation.

deflatePrime

deflatePrime(z_streamp strm, int bits, int value)

The compression operation falls back to software-based compression.

deflateSetHeader(z_streamp strm, gz_headerp head)

Provides the GZIP header information of the specified GZIP stream.

inflateInit2

inflateInit2(z_streamp strm, int windowBits)

Similar to the inflateinit function, but accepts an additional parameter, windowBits.

windowBits

Accepts value of base 2 logarithm. The size of the history buffer (window size) can be in the range 256 bytes - 32 KB. The deflateInit2 function accepts the requested format, but always uses a 32 KB window or history buffer. You can specify values in the following range for the windowBits parameter:

8 - 15: Indicates the zlib format.
24 - 31: Indicates the GZIP format
40 - 47: Indicates automatic header detection.
-8 to -15: Indicates raw deflate format.

The inflateInit2 function accepts the requested format, but always uses a 32KB window or history buffer.

inflateSetDictionary

inflateSetDictionary(z_streamp strm, const Bytef *dictionary, uInt dictLength)

Initializes the decompression dictionary from the specified decompressed byte sequence.

inflateGetDictionary

inflateGetDictionary(z_streamp strm, Bytef *dictionary, uInt *dictLength)

Returns the decompression dictionary that is set in the dictionary parameter.

inflateSync

inflateSync(z_streamp strm)

Skips compressed data during the inflate operation until a possible full flush point is reached. If a full flush point is not reached, the inflate operation skips all the available input data. A full flush point refers to a location in compressed data that was generated when the deflate() function was called with the flush parameter set to the Z_FULL_FLUSH value.

inflateCopy

inflateCopy(z_streamp dest, z_streamp source)

Sets the destination stream as a copy of the source stream.

inflateReset

inflateReset(z_streamp strm)

Resets the compression state but does not free the compression state.

inflateReset2

inflateReset2(z_streamp strm, int windowBits)

The inflateReset2 function is similar to the inflateReset function, but it also allows changing the wrap and window size values. The windowBits parameter is used similarly as it is used in the inflateInit2 function. Size of the history buffer (window size) is static at 32 KB, but the format that is indicated by the windowBits parameter is used for decompression.

inflatePrime

inflatePrime(z_streamp strm, int bits, int value)

Inserts bits in the inflate input stream. This function is used to start the inflate operation at a bit position within a byte. The compression operation falls back to software-based decompression.

inflateMark

inflateMark(z_streamp strm)

Marks location to randomly access data from input stream, which is specified as bit positions in the data stream. Returns two 16-bit values that are upper and lower halves of 32-bit input data.

inflateGetHeader

inflateGetHeader(z_streamp strm, gz_headerp head)

Stores the GZIP header information in the specified header structure.

inflateBackInit

inflateBackInit()

Initializes the internal stream for the decompression operation by calling the inflateBack function. The decompression operation falls back to software-based decompression. This function is supported only at the start of a stream.

inflateBack

inflateBack(z_streamp strm,in_func in,
void FAR *in_desc,out_func out,void FAR *out_desc)

Performs a raw inflate operation by using a call-back interface for input and output data. The decompression operation falls back to software-based decompression. This function must be called only after calling the inflateBackInit function.

inflateBackEnd

inflateBackEnd(z_streamp strm)

Free all memory that is allocated by the inflateBackInit function. The decompression operation falls back to software-based decompression.