ISFC logical link data rates
Figure 1 shows the maximum data rates of the ISFC logical link observed both of its sides during the guest relocation process. The maximum data rates for several ISFC logical link configurations are shown, and for each of them the number of FCTC connections is varied.
Observation
For the cases where up to four FICON channels are used, the write data rates settles for a more or less constant value as soon as four or more FCTC connections are configured per FICON channel. For the cases with five and more FICON channels, the data rates continue to rise as more FCTC connections are used. However, in these cases the limit of sixteen FCTC devices for the ISFC logical link prohibits that four FCTC connections are configured for all of the used FICON channels.
Conclusion
FICON Express4 channels are specified to support 4 Gbit/s link speeds. A practical exploitation is published in the IBM System z9 I/O and FICON Express4 Channel Performance paper (see FICON Express4 Channel Performance): The results shown there were obtained under optimal conditions and show a “full-duplex” FICON channel data rate of 350 Gbit/s for a “large sequential read/write mix”, and a “half-duplex” FICON channel data rate of 330 Gbit/s for a “Large sequential all reads or all writes” use-case.
Subsequently, a respective maximum byte data rate is approximated by dividing the specified bit rate of four Gbit/s by a factor of 10 (considering 8 bits per byte, and assuming 2 bits overhead) – yielding an approximate maximum byte data rate of 400 GB/s.
It is important to realize that for the result of all the shared configurations shown in Figure 1 the FICON channel data transfer capacities are effectively shared between the sending and the receiving LPAR. In other words, for the shared configurations, every FICON channel is simultaneously used for both sending and receiving data.
This simultaneous use is assumed to be a possible reason why in these cases the maximum write byte data rates per FICON channel shown in Figure 1 converge on a value of slightly more than 200 GB/s – which is about half of the maximum data rate per FICON channel that was previously calculated. With the shared configurations, at the same time on the receiving side a read data rate also of slightly more than 200 GB/s is achieved.
Thus, the combined read / write data rate is slightly more than 400 GB/s per FICON channel. This is a very good result that exceeds the expectations expressed in the z/VM performance report (see ISFC Improvements).
On the other hand, for the non-shared configurations, FICON channels are exclusively used by one of the LPARs. Also, in these cases each FICON channel is mostly used for data transfer in one primary direction. These non-shared configurations comprise a situation similar to that when the two LPARs reside on different CECs.
In our tests using the non-shared configurations, the maximum data rates per FICON channel reach values around 355 GB/s per FICON channel if one or two FICON channels are used, and a value of about 348 GB/s per FICON channel if three FICON channels are used. This about 89% (or 87%) of the maximum data rate per FICON channel that was previously calculated, and therefore is in the range of the expectation of 85% expressed in the z/VM performance report.
The ISFC logical link configuration variants with "crossed" FCTC connections achieve data rates that are up to 10% inferior to their "straight" counterparts. One possible explanation for this behavior is that with "straight" FCTC connections– because both LPARs reside on the same CEC and thus use the same FICON channels in one data transfer operation – the data flow could at least to some extent be handled by the FICON channel alone, without being passed through the switch. Regardless, also with "crossed" FCTC connections, the achieved data rates still are slightly above the expectation stated in the z/VM performance report and those expressed in the “IBM System z9 I/O and FICON Express4 Channel Performance” paper.
The slight dent observed for the performance of the non-striped variants of the ISFC logical configurations with three shared FICON channels and three to four "straight" FCTC connections (the two lower blue graphs) is caused by the way z/VM designates FCTC configurations for unidirectional use; this is further substantiated in FCTC connection I/O rates.
Observation
Figure 2 shows the ISFC logical link write data rate as a function of the number of FICON channels that were used. The graph stays linear until up to four FICON channels are used, with a rate of about 204 MB/s per FICON channel. Beyond that, the graph slightly flattens.
Conclusion
Given that the same type of FICON card is used for all FCTC connections (FICON Express4 cards were used in all our test cases), and with the number of FCTC connections per FICON channel configured according to the recommendations given in Recommendations, the data transfer rates of the ISFC logical link scale almost linearly.
The flattening of the graph when more than four FICON channels are used is assumed to result from the limit of sixteen FCTC connections per ISFC logical link. This limit in turn limits the number of FCTC connections at least on some FICON channels to be less than four. However, if less then four FCTC connections are used on a FICON channel, that FICON channel's transport capacity is no longer fully exploited, as will be explained in more detail in FCTC connection I/O rates.