PI68803: APACHE ON Z/OS CPU USAGE INCREASES IN RELEASE 8.5.5.5 OR BEYOND

Fixes are available

APAR status

Closed as program error.

Error description

High CPU usage observed in Apache webserver processes following
upgrade to service level 8.5.5.5 or beyond.
Under the same inbound workload the CPU usage might be 2x or 3x
higher at the later service level.
Dumps show intermittent long periods in z/OS UNIX module
BPXVRCAN which supports asyncio() cancel operations.
High CPU is due to increasing length of the z/OS UNIX Arq
chain, which is maintained on a per process basis. An Arq is
used to represent an individual asycnio() request.
Apache code paths involved with asyncio() activity
were updated to resolve a couple of issues.
(a) a fix to detect and fix one path which might lead to
multiple asyncio() cancel operations for one fd&#247;
(b) a fix to cope with BPX4AIO cancel return code x' 117803FD '
rsn117803FD JRAsyncBadOffset
.
ADDITIONAL SYMPTOMS:
    Storage fragmentation of OMVS high private storage due to
persistent STAR (select status array) control blocks of a very
small length (e.g. x'C8' bytes) in sp229 , key0 storage.
A STAR control block is obtained (along with an ARQ) as part
of an asyncio (Aio#SelPoll) request. The large build-up of
STAR control blocks in OMVS high private and ARQ control
blocks in OMVS dataspace (SYSZBPX1) is due to the Apache
defects being addressed by this APAR.
With these fixes in place it is anticipated that no leaking of
Arq elements should occur.
Note: Diagnosis might be difficult since the periods in
BPXVRCAN occur as a result of an asyncio() cancel operation
and may be in the range of 3-20ms wallclock duration. As a
result it's possible that the system trace in a specific dump
may not capture the period when this occurs.
.
Unix System Services syscalls that involve HFS file operations
cause contention for the OMVS address space's LOCAL LOCK.  When
the ARQ element count is high, each file operation takes
between several seconds and minutes to complete.  Dumps taken
during such slowdowns reveal between 20 and 100 TCBs across
multiple WebSphere App Server, HTTP Server, CICS and DB2
address spaces contending for the OMVS Local Lock.  Call stacks
show many TCBs waiting for HFS I/O to complete. Top java method
could be
int java/io/UnixFileSystem.getBooleanAttributes0(java.io.File)
Top native method could be
libwrappers.so::atoe_lstat+0x88
Note to IBM L2: Please see z-page for further advice

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of IBM HTTP Server on z/OS            *
****************************************************************
* PROBLEM DESCRIPTION: IBM HTTP Server may use elevated CPU,   *
*                      proportional to the age of the httpd    *
*                      process.                                *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
IHS regularly cancels asyncio operations using an AIOCB
allocated from the stack whose "unused" fields were not
explicitly intiailized. This would result in occasional
failures to cancel the outstanding asyncio calls.
If many asyncio calls became orphaned in this way, legitimate
calls to cancel would begin to take more and more CPU time.

Problem conclusion

IHS was updated to always zero out the AIOCB structure used to
perform asyncio cancels.  Additionally, some extra safeguards
were added around reuse of some related datastructures.


The fix for this issue is targeted for the following IHS fixpack
 - 9.0.0.2
 - 8.5.5.11
The issue does not exist in earlier releases 8.0 & 7.0

Temporary fix

Comments

APAR Information

APAR number
PI68803
Reported component name
WAS IHS ZOS
Reported component ID
5655I3510
Reported release
850
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2016-09-07
Closed date
2016-09-27
Last modified date
2017-10-31

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WAS IHS ZOS
Fixed component ID
5655I3510

Applicable component levels

R850 PSY
UP
R90P PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS7K4U","label":"WebSphere Application Server for z\/OS"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"8.5","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
04 May 2022

Tips

PI68803: APACHE ON Z/OS CPU USAGE INCREASES IN RELEASE 8.5.5.5 OR BEYOND

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R850 PSY

R90P PSY

Document Information

Share your feedback

Need support?