Flashes (Alerts)
Abstract
High CPU, OMVS storage growth, or apr_pollset_add failure observed in Apache HTTP server on z/OS
Content
Symptom:
High CPU usage may be observed in Apache HTTP Server for z/OS processes that run for an extended period of time. Under the same inbound workload the CPU usage might be two or three times higher.
Cause:
High CPU is due to increased length of the z/OS UNIX ARQ chain, which is maintained on a per process basis. An ARQ is used to represent an individual asycnio() request.
In some cases the asyncio() cancel operations are failing for various reasons which can lead to a buildup of ARQs. If the ARQ chain gets very long it takes longer to process, resulting in the increased CPU usage, as well as storage fragmentation in OMVS high private storage in some cases.
Diagnosis:
Obtain a dump during the time the problem occurs
--
DUMP COMM=(APACHE HIGH CPU)
R rn,SDATA=(PSA,CSA,LPA,LSQA,RGN,SQA,SUM,SWA,TRT,ALLNUC,GRSQ),CONT
R rn,JOBNAME=(OMVS,<webserver_jobname>),CONT
R rn,DSPNAME=('OMVS'.*),END
The dump may show long periods in z/OS UNIX module BPXVRCAN which supports asyncio() cancel operations, or CML lock contention for OMVS.
Additional symptoms:
- BPX4AIO cancel return code x' 117803FD ' rsn117803FD JRAsyncBadOffset: AioOffset is a negative value
- EDC5112I Resource temporarily unavailable. (errno2=0x11780494): process_socket: apr_pollset_add failure
- OMVS latch contention
How to resolve the problem:
Apply APAR PI68803
PI68803 includes fixes for these APARS:
PI68803 (CPU increase)
PI65855 (HTTPOXY vuln)
PI59561 (MFA base support)
PI66183 (MFA fix)
PI64628 (MSGQ deletion)
The fix for this issue is targeted for the following IHS fixpack
- 9.0.0.2
- 8.5.5.11
- 8.0.0.13
An ifix can be obtained from Fix Central
http://www-01.ibm.com/support/docview.wss?uid=swg24043128
A ++APAR can be requested from Support via a Service Request ticket.
Related Information
[{"Product":{"code":"SSEQTJ","label":"IBM HTTP Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"--","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"8.5.5.9;8.5.5.8;8.5.5.7;8.5.5.6;8.5.5.5;8.5.5.10","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSEQTJ","label":"IBM HTTP Server"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":" ","Platform":[{"code":"PF035","label":"z\/OS"}],"Version":"9.0;8.5.5.9;8.5.5.8;8.5.5.7;8.5.5.6;8.5.5.5;8.5.5.10;8.0","Edition":"","Line of Business":{"code":"LOB45","label":"Automation"}}]
Was this topic helpful?
Document Information
Modified date:
25 September 2022
UID
swg21995140