Configuring optimal page subsystem settings

The settings for the paging subsystem in IBM® Transformation Extender can determine whether a transformation process runs bound by I/O bandwidth or by CPU capacity. These settings can impact system performance.

Behavior

The settings for the paging subsystem within IBM Transformation Extender can determine whether a transformation process runs bound by I/O bandwidth or by CPU capacity. A typical execution time/page count curve has two easily discernible parts, based on performance characteristics.

In a graph where the vertical coordinate measures execution time and the horizontal coordinate measures page count, at the low page count end of the graph, the execution time falls quickly as the page count increases. Here, the transformation is I/O bound. The data access patterns implicit in the map require more information than what is available in memory. Because of this, time spent driving I/O dominates execution time.

As page count increases, execution time decreases into a nearly flat portion of the graph. At that stage, the transformation is bound by the computational power of the executing processor. Increasing the amount of memory available through paging settings will probably not improve performance.

Tuning procedure

Finding the combination of page size and page count that optimizes performance is the goal of the following procedure. There are several steps to follow for tuning paging settings. The settings found for this procedure are specific to the behavior of the map and data used. Because of this, the data should represent future workloads as much as possible.

To tune page settings

Verify that the execution environment is stable. Execution times must be relatively repeatable. Run a few simple preliminary executions and time them to gauge the reproducibility of the timing procedure. If the resulting execution times are not similar to one another, then the execution environment is not stable enough for tuning paging settings. Improve reproducibility by reducing extraneous loads from other applications.
Select initial values for page size and count. The overall page size and count specify the total amount of memory available for paging data and workspace. This total amount of memory is allocated early in the execution process. A failure to allocate paging space will result in an early termination. Therefore, a memory request for C pages of S kilobytes should be expected to succeed, given available storage.

Appropriate values for each setting vary highly on the data access patterns of the map and the design of the type tree. Not only is each setting a factor in execution time, but the overall amount of memory allocated also affects performance. The tuning procedure here will attempt to insulate page size from the effect of changing overall memory
Execute the map and data to be tuned. Note the overall processing time and the processing time spent managing I/O during execution. Sub-optimal paging settings often result in an I/O bound transformation process.
Execute the map and data again with different paging settings. Vary the page size up or down. The page count should be adjusted so that the combined paging memory remains as constant as possible. For example, if you double the page size in the second run, the page count should be halved. Again, note the overall processing time and the processing time spent managing I/O during execution. The page count must still be small enough for the entire paging space to fit in the memory allocated to IBM Transformation Extender by the operating system.
Compare the collected times from both executions after the second execution completes.
Repeat this procedure with various page sizes to find the optimal page size for the map.
After an optimal page size is determined, a similar set of iterations to vary the page count should be performed, keeping the page size fixed.
If an increase in memory drastically reduces performance, it is likely that the overall amount of memory has started to throttle the operating system's virtual memory system. If this occurs, reduce the page count. The threshold at which throttling can occur depends heavily on the application load across the entire system. If a decrease in page count drastically reduces execution time, then the execution was probably I/O bound. Compare the difference in I/O usage between executions or the time used by the operating system. An increase indicates that the proportion of execution time devoted to handling I/O has increased. In this case, increase the page count.

Recent changes

Recent changes in the transformation engine altered the paging behavior of IBM Transformation Extender from release 6.7 onward. These recent changes simplify the tuning process by reducing the CPU overhead required to manage larger page counts. Previously, many maps had a single page count setting that would minimize execution time. Choosing a page count larger than this setting would require more CPU bandwidth. Choosing a smaller page count setting would require more I/O bandwidth, putting the execution into an I/O bound state.