Memory bandwidth

Performance tests with SLES 7 on zSeries showed that the 64 bit kernel had some deficiencies concerning memory bandwidth.

The following charts show some microbenchmarks from the LMbench test suite designed for measuring kernel performance.

We used differently scaled y-axes for better reading.

The y-axis shows the memory throughput in MB per second, the x-axis shows the size of the buffers read/written in KB.

The typical chart should show a cascade: A relatively constant value as long as the operations can be done in first level cache, then a lower value as long as the run can be handled in the second level cache, and finally a value when “real” memory has to be used.

The charts show that in general the bandwidth on the 64 bit system is slightly lower than on the 31 bit system.

Also on 64 bit the cascades happen earlier which indicates that the cache utilization is not as effective as on 31 bit. The libc bcopy aligned test shows some severe degradation in the first level cache range.

Several fixes and performance boosters - including a cache coloring mechanism to specifically handle the cache usage problem - were included in the code. With SLES 8 SP3 we see a completely different results for 64bit can be seen.

The following charts show the test results. The absolute throughput values are higher, because we used different machine types for the tests. (Note: The absolute numbers are higher, but that is mainly due to the different machines types we used for the tests. The cache coloring fix was verified on zSeries 900 with an IBM internal driver.)

A change from IBM eServer zSeries 900 to zSeries 990 did not resolve the cache problem.

The 64 bit numbers are now slightly better than or at least equal to the 31 bit numbers.

The cascading effect appears at the same buffer sizes for 31 bit and 64 bit so the cache utilization efficiency of the 64 bit kernel caught up with the 31 bit kernel.

Back to top