Oracle 11g: multipath daemon - options
The FCP disks devices are another important disk feature. They are integrated into the Linux on System z stack as SCSI devices.
Normalized transactional throughput
They support having many requests open against one disk, but do not provide the advantage DASD devices have with regard to the automated management of multipathing and load balancing between the paths. Setting up multiple paths to one FCP disk can be done easily, but it requires the management of these paths to ensure consistency and to provide a single device to the application. This is provided by a user space tool, the multipath daemon. The multipath daemon can be used with multiple configurations, from a single failover setup to a workload balanced switching between the paths called multibus policy. The latter has as an additional option, the parameter rr_min_io (see Multipath setup), which defines the criteria to switch between the paths based on the amount of requests. The default is to switch to another path after 1000 requests.
Figure 1 shows the performance when the multipath daemon is used in failover mode, and in multibus mode. In the multibus mode the parameter rr_min_io is scaled between 10 and 1000. Due to the observation that in the initial setup with 4 CPUs runs fully utilized, additional CPUs are added.
Observation
With 4 CPUs the system shows no difference in performance. This is because the CPUs are fully utilized. With 6 CPUs the maximum value for rr_min_io is between 100 and 500. The failover mode provides the lowest throughput. For testing purpose a run with 8 CPUs with a rr_min_io value of 100 was done. The improvement was very slight.
Conclusion
With 4 CPUs the throughput was limited by the missing CPU capacity. With 6 CPUs the utilization is still up to 90%, which limits the transactional throughput only slightly, adding even more CPUs provides only a slight improvement.
Total disk throughput (read + write)
FCP disks are driven by the Linux CPUs, they do not take advantage of the System z SAP (system assist processors). The fact that the best rr_min_io setting here is 100 to 500 is probably specific to this type of workload. In terms of disk I/O pattern this means a throughput up to 200MB/sec with small request sizes (8 KB Oracle block size), which is remarkable, but not too high. The expectation is that higher throughput rates, especially with larger block sizes, require smaller rr_min_io values, which means sending less requests to the same path and switching earlier to another path.
Figure 2 shows the disk I/O throughput data reached (reads and writes are summed up).
Observation
The more CPUs available the
higher the disk throughput, the maximum value is 188 MB/sec. With
6 CPUs the throughput in failover mode is 21% lower than with multibus
and rr_min_io=100. The variation within the multibus
setups is much less, the disk throughput with rr_min_io=10 is
only 4% lower than with rr_min_io=100. The remaining
values are identical.
Conclusion
It needs sufficient CPU to drive the FCP disk throughput, and much higher throughput can be reached when the amount of CPU is not limited. Switching between the paths with a multibus setup and after each 100 requests (or later) provided the highest disk I/O bandwidth in our scenario.
CPU cost per transactional throughput
The next interesting question is with regard to the CPU cost required to drive a certain workload. Is failover cheaper, or does frequently switching the path cost additional CPUs. This is shown in Figure 3.
Observation
The CPU cost per throughput is fairly constant for all scenarios, independent of the amount of CPUs. In the setup with 6 CPUs the cost for failover and rr_min_io value of 10 is at 7% slightly higher than for the multibus configurations with the larger rr_min_io values.
Conclusion
Overall, the recommended setup for our workload is a multipath bus setup with the policy multibus and the rr_min_io parameter 100. It provides the highest transactional and disk throughput at the lowest CPU cost.