Results of network block I/O tuning

Edit online

The series of graphics provided in this topic show the results of network block I/O tuning for Spectrum Scale. In particular, the graphics provide a comparison of different MTU sizes.

Figure 1 shows a comparison of using the default MTU size (1492 bytes) and jumbo frames (8192 bytes) either SD or NSD. The performance metrics were the average ICN transaction response time at the highest load level.
Figure 1. Average ICN response time for different MTU sizes

This graphic provides an overview of how the average ICN response time varies for different MTU sizes
As shown in Figure 1, the Spectrum Scale cluster configuration SD did not significantly benefit from the use of jumbo frames.
  • The average transaction response time dropped to below 300 ms with jumbo frames. For NSD, the improvement with jumbo frames was much larger.
  • The average transaction response time dropped to 300 ms, and was then at the same level as Spectrum Scale SD with default MTU.
  • When looking at the hypervisor CPU load (z/VM® total load), there was also a CPU cost savings with the usage of jumbo frames.
Figure 2 shows the CPU loads for the z/VM hypervisor.
Figure 2. CPU cost savings (hypervisor view) for Spectrum Scale NSD when using jumbo frames

This graphic provides an overview of the CPU cost savings (from a Hypervisor view) for Spectrum Scale NSD when using jumbo frames
As shown in Figure 2:
  • By using jumbo frames for the complete SUT, the hypervisor load (total z/VM load) was reduced by 1.7 IFL processors.
  • The CPU cost savings was the result of less networking overhead with the usage of jumbo frames.
  • The processing for the z/VM virtual network (VSWITCH based) was done by IFL processors.
  • When using jumbo frames, additional payload data fits into a single network packet. As a result, the Linux™ network packet rate dramatically went down.
Figure 3 further shows the results of using jumbo frames. It shows Linux network packets rates (packets/sec) and average packet size (bytes/packet). The listed rates and sizes were for one ECM node in the ECM cluster. However, the reported metrics would quadruple if the entire ECM cluster was considered.
Figure 3. Network jumbo frames – network packet rates and sizes on ECM nodes

This graphic provides an overview of how the network packet rates and sizes on ECM nodes varied when using Jumbo frames)
As shown in Figure 3:
  • The packet rates and sizes were shown for the default MTU size (1492) and for jumbo frames (8192). Furthermore, the network packet rate was a stacked number for the received (rx) and transmitted (tx) packets displayed as a single bar.
  • The packet sizes for the rx and tx packets are shown as two grey bars next to the packet rate bars.
  • With jumbo frames enabled, the Linux network packet rate was reduced to 30,000 packets/sec compared to more than 70,000 packets/sec when the default MTU was used. This corresponded to a packet rate reduction of more than 60%.
  • If the received and transmitted packet size metrics for the default MTU are examined, the network packets were almost completely using the available payload (packet size vs. MTU size). With jumbo frames, the network packet sizes were 2.5 times to 3 times larger.
  • The payload data for a single network packet was larger. As a result, less network packets were needed to transport the same amount of user data.

Only enabling jumbo frames for z/VM virtual networking

Jumbo frames are sometimes not available throughput an entire LAN infrastructure. Virtual networking for the z/VM hypervisor was provided with a Virtual Switch (VSWITCH) for the SUT. Therefore, network traffic between the z/VM virtual machines was inside the scope of the hypervisor.

Since all Spectrum Scale network block I/O was between virtual machines of the same z/VM, enabling jumbo frames for these virtual machines did not cause any difficulties. The workload driver machine running on Linux on x86_64 was kept at the default MTU (1500) for this side study.

Therefore, any network traffic between the virtual machines of the z Systems™ server and the external workload machine could not use jumbo frames.

In Figure 4, three measurements are compared:
  • The left bar shows the average ICN transaction response time when jumbo frames were enabled for the complete SUT (including the x86_64 workload driver machine).
  • The bar in the middle shows the response time when jumbo frames were enabled only inside z/VM.
  • The rightmost bar shows the default networking MTU (1492) throughout the SUT.
Figure 4. Jumbo frames for virtual networking only – average ICN response time

This graphic provides an overview of how the average ICN response times varied when using Jumbo frames for virtual networking only

There was only a 5% increase of the response time when jumbo frames were enabled for z/VM only compared to jumbo frames for the complete SUT. This was still a significant improvement in comparison to the response time with the default network MTU.

Conclusion

Even if the entire network infrastructure (that is, network switches or other servers) does not support jumbo frames, it can still be beneficial to enable jumbo frames for z/VM virtual networking only.

Jumbo frames can help to improve throughput or response times when enabled for Spectrum Scale network block I/O.