PI68803:Apache on z/OS CPU usage increases in release 8.5.5.5 or beyond

Download Description

Note: The interim fix for this APAR has been superseded by both the PI73984 interim fix and the even newer PI82481 interim fix. It is recommended that you use the latest available APAR interim fix that is applicable to your version.

PI68803 resolves the following problem:

ERROR DESCRIPTION:
High CPU usage observed in Apache webserver processes following
upgrade to service level 8.5.5.5 or beyond.
Under the same inbound workload the CPU usage might be 2x or 3x
higher at the later service level.
Dumps show intermittent long periods in z/OS UNIX module
BPXVRCAN which supports asyncio() cancel operations.
High CPU is due to increasing length of the z/OS UNIX Arq
chain, which is maintained on a per process basis. An Arq is
used to represent an individual asycnio() request.
Apache code paths involved with asyncio() activity
were updated to resolve a couple of issues.
(a) a fix to detect and fix one path which might lead to
multiple asyncio() cancel operations for one fd
(b) a fix to cope with BPX4AIO cancel return code of 0x117803fd
With these fixes in place it is anticipated that no leaking of
Arq elements should occur.
NB. Diagnosis might be difficult since the periods in
BPXVRCAN occur as a result of an asyncio() cancel operation
and may be in the range of 3-20ms wallclock duration. As a
result it's possible that the system trace in a specific dump
may not capture the period when this occurs.
Note to IBM L2: Please see z-page for further advice

PROBLEM SUMMARY:
IHS regularly cancels asyncio operations using an AIOCB
allocated from the stack whose "unused" fields were not
explicitly intiailized. This would result in occasional
failures to cancel the outstanding asyncio calls.
If many asyncio calls became orphaned in this way, legitimate
calls to cancel would begin to take more and more CPU time.

PROBLEM CONCLUSION:
IHS was updated to always zero out the AIOCB structure used to
perform asyncio cancels. Additionally, some extra safeguards we
added around reuse of some related datastructures.

Note: This interim fix also includes fixes for the following APARs:
PI59561 :Add Pre/Post password hooks to mod_authnz_saf (MFA base support)
PI64628 :IBM HTTP Server on z/OS is deleting the wrong IPC message queue
PI65855 :CVE-2016-5387 (HTTPOXY) for IBM HTTP Server
PI66183 :When MFA is configured, SAFRunAs fails with a permission error

The fix for this issue is targeted for the following IHS fixpack
- 9.0.0.2
- 8.5.5.11
- (not applicable to earlier releases)