zHyperLink Write Support

I/O latency as viewed by the application is more than just I/O service time. It also includes interrupt processing time, time waiting to get dispatched on a CPU after an I/O has completed, and delays caused by having to reload the CPU cache. Traditional I/O to DASD is asynchronous, which means that the application requests a record, the operating system initiates an I/O request, and the application is suspended until the I/O completes and the application is redispatched on a CPU. zHyperLink I/O is synchronous, which means that the CPU spins waiting for the I/O request to complete. This eliminates the delays normally associated with asynchronous I/O.

I/Os that are good candidates for zHyperLink are those where the application is suspended until the I/O completes. Two examples are Db2 synchronous database reads that occur when the application requests a record that is not in the buffer pool and Db2 log writes issued at transaction commit time. I/Os that are not good candidates for zHyperLink are reads used for prefetching or read-ahead and delayed writes because both are performed asynchronous to the execution of the application.

zHyperLink write requests have an additional requirement: the write I/Os must follow a log write pattern, which means that the writes must always go forward in the data set without skipping any records. The same record may be written multiple times, but the writes never go backwards. When you are finished with the last record in the data set, you can restart at the beginning of the data set.

In 2019, zHyperLink write support was introduced for simplex and local Metro Mirror environments (see Figure 1). In Metro Mirror configurations all DS8000s must be within 150m of the Z server and connected via zHyperlink.

In 2020 for z/OS V2R3 and above, zHyperLink write support was provided for DS8000 Global Mirror (asynchronous mirroring) configurations (see Figure 2 for an example). This does not include support for Extended Remote Copy (XRC).

Db2 for z/OS currently exploits zHyperLink writes for active log data sets. Media Manager support for dual logging provides the ability for Db2 to issue zHyperLink writes to two active log data sets simultaneously.

Figure 1: zHyperLink writes in a Metro Mirror environment.

Figure 2: zHyperLink writes in a Global Mirror environment.

zHyperLink is a point-to-point connection between the IBM Z processor and a DS8000. The maximum distance is 150 meters. To use zHyperLink, you need one or more zHyperLink Express features (cards) in the IBM Z processor. Each feature supports up to 2 ports and up to 16 features (32 ports) can be installed.

On the DS8000 side, each zHyperLink connection plugs directly into a port in the I/O enclosure; a special host adapter card is not required. Up to 2 connections per I/O enclosure is supported. The maximum number of connections supported is model dependent and is based on the number of enclosures and cores. zHyperLink connectivity is also required to the secondary for write support in Metro Mirror environments, which means both primary and secondary must be within 150m of the IBM Z processor.

FICON connectivity to the DS8000 is still required for initialization for performing I/Os that are not eligible for zHyperLink and as a fallback if the zHyperLink I/O fails. Using zHyperLink does not require you to reduce the maximum number of FICON channel paths configured for a device.

In the IBM Z I/O configuration, define the PCIe function definitions (PFIDs) for the zHyperLink ports. The association between the zHyperLink port and DS8000 is discovered dynamically. Define 4 PFIDs per zHyperLink port for each logical partition that is sharing the DS8000.

Find the PTFs for zHyperLink by searching for Fix Category IBM.Function.zHyperLink, APAR keyword HYPERL/K.

On a z/OS LPAR, zHyperLink is disabled by default. To enable zHyperLink write processing during IPL time, include the following in the IECIOSxx parmlib member:

ZHPF=YES, if zHPF is not already enabled on the LPAR.

ZHYPERLINK,OPER=WRITE or OPER=ALL. If ALL is defined, both read and write processing is enabled.

zHyperLink write can also be enabled dynamically on a z/OS LPAR using these z/OS commands:

SETIOS ZHPF=YES, if zHPF is not already enabled on the LPAR.

SETIOS ZHYPERLINK,OPER=WRITE or OPER=ALL.

To verify if zHyperLink support is enabled, issue:

• DISPLAY IOS,ZHYPERLINK to display whether zHyperLink is enabled for writes for the system (see Figure 3).

• DISPLAY M=DEV(devno) to display whether zHyperLink is enabled for writes for a device (see Figure 4).

• DISPLAY M=DEV(devno),ZHYPERLINK to display whether zHyperLink is enabled for writes for a device. If zHyperLink is disabled, the reasons why zHyperLink is disabled are displayed.

• DISPLAY M=CU(cuno) to show whether zHyperLink is enabled for writes at the control unit level and to show the state of the PFIDs (see Figure 5).

Figure 3: DISPLAY IOS,ZHYPERLINK command output.

Figure 4: DISPLAY M=DEV(devno) command output.

Figure 5: DISPLAY M=CU(cuno) command output.

• zHyperLink write support is available on Db2 12 for z/OS.

To enable zHyperLink, use the Interactive Storage Management Facility (ISMF) interface to change the zHLink Write parameter of the storage class used to allocate the Db2 active log data sets to YES from the default of NO. Use the VARY SMS command to override the storage class setting for SMS managed data sets or to allow zHyperLink writes to be used for non-SMS managed data sets. Note: The VARY SMS command change does not persist across an IPL.

To exploit zHyperLink support for Db2 active log datasets, the ZHYPERLINK Db2 subsystem parameter must be set to either ACTIVELOG or ENABLE. This can be dynamically updated while Db2 is running by issuing the Db2 SET SYSPARM command.

• zHyperLink write support for synchronous mirroring is available on the DS8880 and DS8900.

• zHyperLink write support for asynchronous mirroring is available only on the DS8900.

To enable zHyperLink, logon to DS8000 HMC as an administrator, select Settings and then System to see the list of supported licensed functions. The zHyperLink Function shows the options to enable I/O Write and I/O Read (see Figure 6). Enabling zHyperlink writes on the DS8000 will result in a quiesce/resume of the DS8000 servers similar to a code load, and so while this is non-disruptive, it would typically be scheduled for a quiet time.

Figure 6: Enabling zHyperLink on DS8000.

For Metro Mirror environments, the devices on which the Db2 log data sets reside must be in the duplex state, in a HyperSwap configuration, and zHyperWrite must be enabled. In the IBM test lab, the solution was tested when the devices were in a simplex and the following mirroring configurations:

• Single site

  • Simplex

• 2-site

  • Metro Mirror (MM) (see Figure 7)

  • Global Mirror (GM) (see Figure 8)

• 3-site

  • Multi-Target (MT) MM/MM

  • Multi-Target MM/GM (see Figure 9)

  • MGM (see Figure 9)

Figure 7: Two Site Metro Mirror environment.

Figure 8: Two Site Global Mirror environment.

Figure 9: Three Site Multi-Target MM/GM or MGM environment.

zHyperLink writes provide lower latency than zHPF in all replication configurations. Figure 10 compares the performance of zHyperLink writes to zHPF writes with no mirroring and with different mirroring environments (Metro Mirror, Global Copy, Global Mirror). These results were achieved with 26-meter zHyperLink connections using a workload that simulates Db2 log writes that were 4K in size. Note: In this test, zHyperWrite was not enabled when using zHPF in a Metro Mirror environment so the response time shown is higher than it would have been with zHyperWrite enabled. With zHyperWrite, the response time for Metro Mirror would be similar to no mirroring.

Figure 10: zHyperLink vs zHPF response time with different mirroring configurations.

Figure 11 compares zHPF to zHyperLink performance in a Global Mirror environment with a mixed random/sequential read and write workload and using a program that emulates Db2 log writes. zHyperLink reads were not enabled for this test. There was a 4x reduction in log write response time with zHyperLink. Overall, response time showed only a small improvement because reads made up a majority of the I/Os. In addition, this test showed that the success rate for zHyperLink writes was unaffected by Global Mirror and the consistency group formation time, which affects the Recovery Point Objective (RPO), was not affected by zHyperLink writes (1.1 seconds with and without zHyperLink).

Figure 11: zHyperLink write vs FICON with Global Mirror consistency group.

RMF shows synchronous I/O performance data at the DASD device and PCIE function level. Figure 12 shows an example of the RMF Synchronous I/O Device Activity report, which contains the zHyperLink read and write activity as well as the asynchronous I/O activity.

Figure 12: RMF synchronous I/O device activity report.

The DISPLAY LOG command for Db2 for z/OS shows if zHyperLink writes is enabled for the active logs and whether the last log write for the active log data sets used zHyperLink (see Figure 13).

Figure 13: Db2 DISPLAY LOG command output.

Alternatively, DFSMS collects detailed zHyperLink write statistics and makes them available with the SMF type 42 subtype 6 records and the DISPLAY SMS,DSNAME(dsn),STATS(ZHLWRITE) command (see Figure 14). The statistics are provided at both a Db2 request level as well as the device level.

Figure 14: DISPLAY SMS command output.

Omegamon XE for Db2 Performance Expert shows the average elapsed time for class 3 suspensions for synchronous I/O log writes. The TIME/EVENT for SYNCHRONOUS LOG WRITE I/O shows the benefit of using zHyperLink writes for Db2 active logs when compared to reports prior to enabling zHyperLink writes (see Figure 15).

Figure 15: Omegamon XE for Db2 Performance accounting report - long.

Tariq Hanif is a Senior Software Engineer in the z/OS I/O Supervisor Team. In z/OS, his focus areas of testing are Parallel Sysplex, Storage Replication Solutions and Disaster Recovery.

Brian Lee is a Software Developer for z/OS DFSMS Media Manager.

Dale Riedy is a Senior Technical Staff Member with z/OS I/O Supervisor Design and Development.

We would like to thank Nick Clayton, Distinguished Engineer - Enterprise Storage Development, for reviewing and providing comments on this article.

Rita Beisel contributed to the editorial review of this article.