File system I/O workload

Workload analysis

Figure 1 shows the relative throughput of the medium level file system I/O workload for both the direct I/O and the page-cached variant, when these were executed as part of the combined workload set within a variety of differently constrained z/VM configurations.

Figure 2 shows the relative throughput of the high level file system I/O workload for both the direct I/O and the page-cached variant.

A general description of the elements shown in Figure 1 and Figure 2 is given in General description of relative throughput diagrams.

Observations:

The dependency of the throughput on the amount of z/VM real memory remained in the area of a few percent, except when z/VM had 20 real CPUs configured. In this case, and when z/VM real memory was 64 GiB or less, the throughput of the direct I/O variant increased in spite of z/VM real memory being more constrained. At the same time, the throughput of the page-cached variant decreased respectively. In fact, the throughput achieved by the high level workload in this case even slightly exceeded that of the unconstrained case.

Conclusions:

The file system I/O workload did not benefit from using the page-cache. When z/VM real memory was significantly constrained, but processing power was not constrained, the direct I/O variant apparently even reclaimed throughput at the expense of the page-cached variant.

A possible explanation for this at first unexpected behavior might be higher virtual memory access latencies. As page-cache pages were dirtied by fio writes, but were not immediately written to the target file by Linux™, the access rate on such pages might have appeared low to z/VM. When under real memory pressure, z/VM very soon might have decided to page-out such pages, in order to free corresponding page frames for use by other memory consumers. However, later, when Linux again accessed such pages in order to write them to the I/O file, z/VM first had to obtain (or reclaim) page frames, and fetch previously paged out pages.

A central point in favor of the direct I/O variant is that it uses only a few and comparatively small data transfer buffers. However, these buffers are used very intensely, preventing that respective virtual memory is paged-out by z/VM. Consequently, the workload is not hindered by memory access latencies and can continue to make use of CPU resources.

Another factor working in favor of the direct I/O variant is the fact that the target I/O resources for both file system I/O workload variants were located on the same storage server, and were connected by the same SAN resources (see Hardware). As the server that was processing the page-cached variant became more and more limited in accessing pages involved in page-cache I/O operations, at the same time I/O resources became relatively better available for the server that was processing the direct I/O variant.

On the other hand, if in addition the processing power was limited for z/VM and implicitly also for the virtual systems running under z/VM, then, as a result, the process of dirtying pages as well as any other virtual system activities was retarded . Insofar, the effects resulting from constrained z/VM real memory were moderate, the more the processing power became limited, or, in case of very constrained processing power, were not observed at all.

Memory usage and paging workload analysis for CID 64/20

As with the database BI workload, here the detailed memory usage and workload analysis is only presented for the z/VM configuration with 64 GiB real memory and 20 real CPUs (CID 64/20).

Figure 3 shows the average page distribution and the average page read rates for both levels of the page-cached variant of the file system I/O workload, when running as part of the combined workload set in a z/VM CID 64/20 system configuration.

A general description of the elements shown in Figure 3 is given in General description of page distribution and page read rate diagrams.

Observations for the page-cached variant of the file system I/O workload:

The page-cached variants of the file system I/O workload exhibited rather large amounts of instantiated memory. For the medium level workload, on average about 66 % of the instantiated memory was backed by resident memory, and for the high level workload, this percentage was even 71 %. This resident memory accounted for 8.5 % (medium level) and 14.5 % (high level) of the z/VM DPA. The average amounts of instantiated memory observed during the reference execution were significantly higher than those observed with the z/VM CID 64/20 configuration.

The page-cached variants of the file system I/O workload averaged at about 2.3 % (medium level) and 3.4 % (high level) of z/VM's page read operations. The averaged fractions of resident memory being paged in were about 0.4 % for the medium level, and about 0.3 % for the high level workload.

Figure 4 shows the average page distribution and the average page read rates for both levels of the direct I/O variant of the file system I/O workload. Note that the left scale for memory is different from that used in Figure 3.

A general description of the elements shown in Figure 4 is given in General description of page distribution and page read rate diagrams. Note that the memory scale in Figure 3 and Figure 4 is different from that used in previous page distribution and page read diagrams.

Observations for the direct I/O variant of the file system I/O workload:

The direct I/O variants of the file system I/O workload exhibited rather small amounts of instantiated memory. For the medium level workload, on average, about 3.6 % of the instantiated memory was backed by resident memory, and for the high level workload this percentage was 9.8 %. These resident memory sizes accounted for 0.05 % (medium level) and 0.14 % (high level) of the z/VM DPA. Again, the average amounts of instantiated memory observed during the reference execution were significantly higher than those observed with the z/VM CID 64/20.configuration.

The direct I/O variants of the file system I/O workload did not cause any significant z/VM paging activities.

Conclusions for both variants of the file system I/O workload:

As expected, when using the page-cache for the file system I/O workload, much higher memory consumption results. In this case, when memory constraints at the z/VM level result in respective paging activities, memory access delays are imposed at the virtual system level.

On the other hand, the direct I/O variants of the file system I/O workload used only small amounts of memory. As memory constraints at the z/VM level became more distinct, the direct I/O variants might have profited from other workloads not reaching their full processing capacities because they were more affected by memory access latencies.

The fact that the average amounts of instantiated memory observed within CID 64/20 were much smaller than those observed during the reference execution could be explained as follows:

Any test case execution went through an initial phase, during which the software within each virtual system was initialized for the test. During this phase, typically memory intense tasks had to be performed. After this initial phase, Linux flagged unused parts of the virtual memory as unused (see also Interactions between virtual memory consumers and providers).

When executing within a configuration that is not memory constrained – such as during the reference execution – CP is not under pressure to reassign real memory backing unused virtual memory for other purposes. Thus, in this situation, the unused virtual memory could remain part of the instantiated memory.

However, when executing within a memory constrained environment such as within CID 64/20, CP removes these virtual memory parts from the virtual system's instantiated memory as soon as it detects that parts of virtual memory were declared unused by virtual systems. CP can then reassign the backing real memory for other purposes. Of course, this approach is preferable over stealing the backing real memory that is in use, because used virtual memory would have to be paged-out before the backing real memory could be reassigned.

With respect to the allocation and use of virtual memory there is a major difference between the page-cached variant and the direct I/O variant of the workload:

• Page-cached I/O variant

  With this variant the data relating to file I/O operations is cached in the page cache. The observed size of the page cache was about 6 GiB for the medium level variant, and about 10 GiB for the high level variant. Note that the page cache virtual memory requirements are in addition to other memory usage, such as that for application buffers or for device driver buffers.

  The lower amount of instantiated memory when compared to that in use during the reference execution is caused by the significantly reduced throughput that in turn caused a lower need for page cache space.

• Direct I/O variant

  With this variant the data relating to file I/O operations is directly transferred between application buffers and device driver buffers, without using the page cache. This results in a very small memory footprint. However, even here the observed size of the page cache was about 300 MiB. In this case the use of the page cache was imposed by other elements maintained by the Linux operating system, such as shared libraries or processes.

The virtual systems assigned for executing the file system I/O workload are examples of memory oversized virtual systems. Nevertheless, this does not cause a significant waste of real memory, because real memory backing unused virtual memory can be reassigned for other purposes when needed.