Are you experencing CPU performance problems with V5.1.1 or V5.2 after enabling zIIP?

Body

You are experiencing CPU performance issues. A review of the system reveals that the Connect:Direct for z/OS (CDZ) DTF was building a large number of enclaves (somewhere in the 20,000 range) and not releasing them.

With the introduction of zIIP support in CDZ V5.1.1 it was learned that an enclave (a z/OS construct that allows a unit of work or transaction to be assigned a goal by the z/OS Workload Manager (WLM)) has to be built for the process to do zIIP processing. The enhancement that added zIIP support had changed code to bypass the code that would have released the enclave when a process failed. If the failure was for a reason that would cause the process to be requeued and retried (loss of connectivity) a new enclave would be built for each retry of the process.

At some point in V5.1.1 it was decided to allow the customers to turn zIIP support on or off on a process by process basis. By allowing this the decision was made that CDZ would have to build an enclave for every process that established a session regardless of whether the process was outbound or inbound. The problem that was discovered was that when zIIP support was added to CDZ that the code to clean up the enclaves after a failure was being bypassed. If the process was requeued for a restart a new enclave was built. We have had several customers that have seen 20,000+ enclaves associated with the DTF hanging around when they start experiencing performance problems. The performance affects the entire LPAR. There is no command to release these enclaves, the only way to get rid of them when they build up to the excessive level is to bounce the DTF.

This was further compounded in V5.2 when support was added that made zIIP the fallback for compressing and decompressing datasets if zEDC is not available. Since you don't know whether a process is going to enable or disable zIIP in the script or need to fallback to zIIP an enclave has to be built for every process that is executed, outbound or inbound.

Then if you go to having problems with processes failing, being requeued, and restarted you can get a lot enclaves. In fact if you get into a Intelligent Retry loop these enclaves can build up fairly quickly.

There is a PTF for this issue, UI36892. The problem here is that UI36302 is a prerequisite for this fix. Most customers are most likely not going to see this issue. They are not aware of this issue until they begin to have system performance issues and discover that there is a VERY large number of enclaves associated with the DTF. The number of enclaves can be monitor with the ENC command from the SDSF command line.

In most cases the customers we have seen with the excessive number of enclaves is because they have a very high rate of processes failing with loss of connectivity for any kind of network issue. If they are unable to apply any of the maintenance to resolve this issue they will need to take steps to monitor their enclaves and see if there is anything they can do to reduce the number of loss of connectivity failures.

If running CDZ V5.1.1 apply PTF UI24173.

If running CDZ V5.2 apply PTF UI36892

Workaround: There is no command to release the enclaves. You will need to bounce the DTF to release the enclaves. You can monitor the number of enclaves by issuing the line command ENC in SDSF.