Oracle 11g: using HyperPAV devices
When running databases with high I/O requirements on DASD devices, especially large disks, a significant degradation can appear by the subchannel busy contention.
Impact on transactional throughput and disk I/O throughput
When one request is sent to a disk (=subchannel), the logical device subchannel is busy and does not accept further requests. In the past the use of PAV alias devices was the recommended way to deal with that issue. But it does have some disadvantages, as it requires a certain amount of PAV devices per disk, which can heavily increase the amount of disk devices in the system, and it requires the setup of the multipath daemon. This adds as a minimum code to the path length and management overhead. A much more convenient solution is the use of HyperPAV. This uses a pool of alias devices for all DASDs from one LCU (LSS). The devices need only to be configured on the storage server and enabled in Linux. Everything else including load balancing is handled by the kernel. An additional advantage is that the same HyperPAV alias devices can also be used from other LPARs and from other systems.
The important question is, how many alias devices are needed and what is the impact on transactional throughput. Figure 1 shows transactional and disk throughput, when scaling the amount of HyperPAV devices for our workload. We used two LCUs (one from each internal server of the DS8000), the chart specifies the sum of the HyperPAV devices from both LCUs.
Observation
When increasing the amount of HyperPAV aliases the disk I/O throughput increases heavily in the first step. But with further scaling steps it increases at a much lower rate and then reduces slightly. The transactional throughput follows the disk throughput, but the reduction at the end of the scaling is only very slight. The transactional throughput increases with 20 HyperPAV aliases by more than 40%.
Conclusion
The usage of HyperPav devices can improve the transactional and the disk I/O throughput significantly. Be aware that with the switch to Oracle 11g the disk I/O rate was dramatically reduced. The expectation is that the impact increases with increasing disk I/O requirements. It is also interesting that the amount of required HyperPAV devices is for this workload with 20 devices relatively small. Too many HyerpPAV devices might limit the positive impact. The big advantage of HyperPAV with respect to PAV is the really simple handling, it just needed to be enabled in Linux, all the multipathing and workload balancing is done by the Linux kernel.
Impact on the total execution times of the DASD requests
To show what happens when HyperPAV devices are added, the following Figure 2 shows the impact of the amount of HyperPAV devices on the run times of the DASD I/O requests, as reported from the dasd statistics in /proc/dasd/statistics. It depicts the percentage of I/O requests from all requests of a run with an I/O time within a certain period.
Observation
Without HyperPAV there is a peak around 1-2 milliseconds. With 10 alias devices a peak with very short running requests already appears at below 0.5 milliseconds. This peak does not change with more alias devices, but the peak at 1-2 milliseconds reduces the height and disperse to longer time values. The overall number of requests increases, according to the throughput.
Conclusion
Without HyperPAV, most DASD requests have four other requests in the device driver queue before themselves, which would provide a delay of up to 2 ms (4 x 0.5ms) for each request. This contention is already significantly reduced with 10 alias devices already. But the tendency to have longer running requests with more alias devices indicates that the pressure on the storage server causes it to react with longer services times for the additional requests.
Impact on duration times of the DASD requests in the SAN network
To analyze the behavior of the SAN and the storage server Figure 3 shows the histogram of the times from the startsubchannel command (SSCH) until the execution of the request is signaled via interrupt.
Observation
From no HyperPAV to 10 HyperPAV devices the amount of requests just increases, with 20 alias devices the shape starts changing, it goes down at the short times end and increases with longer time values, which continues with 40 alias devices.
Conclusion
The time from issuing the start subchannel command until the interrupt signals the availability of the responses, consists of the time spent in the SAN and the response times from the storage server. Typically the time spent in the SAN is very short, that means the major contribution comes from the storage server. Therefore this chart shows that with the increasing amount of requests the storage server answers with longer response times for some of the requests. This is a typical indication of pressure on the storage server side, even when the times with 4 ms are still very short. Comparing these runtimes with the total times shown in figure 14 shows that with 40 HyperPAV devices there are a large amount of requests with total times of (from enter to the exit of the DASD layer) between 2 and 8 ms. This indicates that in average each request now has to wait for at least one request to finish, because of the longer storage server response times.
The overall conclusion is that HyperPAV is a very good feature for increasing the I/O bandwidth with DASD devices. 10 to 20 alias devices per LCU was a good number for this workload. But another important finding is that too many devices lead then again to a degradation, even if it is moderate. Here monitoring the response times of the storage server (for example with the DASD statistics) can help to decide what is a good number. It should also be mentioned that the administration effort to use HyperPAV in Linux is minimal, which makes it a very convenient tool.