PH69144: APACHE CXF CLASS - DELAYEDCACHEDOUTPUTSTREAMCLEANER

APAR status

• Closed as program error.

Error description

• JAX-RS. In apache CXF class - DelayedCachedOutputStreamCleaner
  threads keep growing. See
  alsohttps://issues.apache.org/jira/browse/CXF-9171.  Update to
  DelayedCachedOutputStreamCleaner file may be required.
  

Local fix

Problem summary

• ****************************************************************
  * USERS AFFECTED:  All users of IBM WebSphere Application      *
  *                  Server Liberty Core using apache cxf        *
  ****************************************************************
  * PROBLEM DESCRIPTION: The jvm threads keep growing after      *
  *                      upgrading to liberty 25.0.0.5 from      *
  *                      24.0.0.12. The class responsible for    *
  *                      this behaviour:                         *
  *                      DelayedCachedOutputStreamCleaner.       *
  *                      When researching the problem we         *
  *                      noticed following change that           *
  *                      introduced the problem: #31097.        *
  *                      This was done for liberty 25.0.0.4      *
  *                      and is the version where the problem    *
  *                      starts occurring. No threads are        *
  *                      being freed even if there are no        *
  *                      requests being send.                    *
  ****************************************************************
  * RECOMMENDATION:                                              *
  ****************************************************************
  The jvm threads keep growing after upgrading to liberty
  25.0.0.5 from 24.0.0.12. The class responsible for this
  behaviour: DelayedCachedOutputStreamCleaner. When researching
  the problem we noticed following change that introduced the
  problem: #31097.
  This was done for liberty 25.0.0.4 and is the version where
  the problem starts occurring. No threads are being freed even
  if there are no requests being send.
  Thread count Trace
  base_REST_request_total{class="com.demo.rest.VoertuigenResource"
  ,method="geefCcgVoorVoertuig_java.lang.String"} 3372
  TYPE base_thread_count gauge
  HELP base_thread_count Displays the current number of live
  threads including both daemon and non-daemon threads.
  base_thread_count 3873
  Trace showing issue class
  LOCKS subcomponent dump routine
  NULL ===============================
  NULL
  1LKPOOLINFO Monitor pool info:
  2LKPOOLTOTAL Current total number of monitors: 1287
  NULL
  1LKMONPOOLDUMP Monitor Pool Dump (flat & inflated
  object-monitors):
  2LKMONINUSE sys_mon_t:0x00007F876C0013B8 infl_mon_t:
  0x00007F876C001438:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E41EC8C8:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "Executor Service Control Timer"
  (J9VMThread:0x0000000000728F00)
  2LKMONINUSE sys_mon_t:0x00007F876C001518 infl_mon_t:
  0x00007F876C001598:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E990DD98:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "DelayedCachedOutputStreamCleaner"
  (J9VMThread:0x00000000036B1B00)
  2LKMONINUSE sys_mon_t:0x00007F876C001678 infl_mon_t:
  0x00007F876C0016F8:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E3961E08:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "Timer-1" (J9VMThread:0x0000000000364D00)
  2LKMONINUSE sys_mon_t:0x00007F876C0019E8 infl_mon_t:
  0x00007F876C001A68:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E40B4738:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "Timer-4" (J9VMThread:0x0000000000602A00)
  2LKMONINUSE sys_mon_t:0x00007F876C002DD8 infl_mon_t:
  0x00007F876C002E58:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E3A55D30:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "Timer-2" (J9VMThread:0x0000000000368A00)
  2LKMONINUSE sys_mon_t:0x00007F87D9511108 infl_mon_t:
  0x00007F87D9511188:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E95BA670:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "DelayedCachedOutputStreamCleaner"
  (J9VMThread:0x000000000357D800)
  2LKMONINUSE sys_mon_t:0x00007F87D95111B8 infl_mon_t:
  0x00007F87D9511238:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E3A42B60:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "Timer-0" (J9VMThread:0x000000000026F600)
  2LKMONINUSE sys_mon_t:0x00007F87D95122E8 infl_mon_t:
  0x00007F87D9512368:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E7739A80:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "DelayedCachedOutputStreamCleaner"
  (J9VMThread:0x0000000001BC9700)
  2LKMONINUSE sys_mon_t:0x00007F87D9512918 infl_mon_t:
  0x00007F87D9512998:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E99E1160:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "DelayedCachedOutputStreamCleaner"
  (J9VMThread:0x0000000003703A00)
  2LKMONINUSE sys_mon_t:0x00007F874802AB18 infl_mon_t:
  0x00007F874802AB98:
  3LKMONOBJECT java/util/TaskQueue@0x00000000E97083F8:
  3LKNOTIFYQ Waiting to be notified:
  3LKWAITNOTIFY "DelayedCachedOutputStreamCleaner"
  (J9VMThread:0x00000000035C3E00)
  2LKMONINUSE sys_mon_t:0x00007F874802ADD8 infl_mon_t:
  0x00007F874802AE58:
  3LKMONOBJECT
  org/eclipse/osgi/framework/eventmgr/EventManager$EventThread@0x0
  0000000E407CAD8:
  

Problem conclusion

• Expected behaviour is that during the REST requests firing,
  the base_thread_count should show only a spike of 20-30
  irrespective of the number of requests fired. Once the request
  firing is stopped, the base_thread_count should come back to
  initial value.
  Apache cxf team reported the issue
   https://issues.apache.org/jira/browse/CXF-9171  for the
  thread growing problem and fix is done in apache cxf class -
  DelayedCachedOutputStreamCleaner. Backported those changes to
  Liberty and could see that the issue got fixed. Testing done
  by creating a new fat test for this issue.
  
  The fix for this APAR is targeted
  for inclusion in 26.0.0.1
  

Temporary fix

Comments

APAR Information

• APAR is sysrouted FROM one or more of the following:

• APAR is sysrouted TO one or more of the following:

Fix information

• Fixed component name

  WAS LIBERTY COR

• Fixed component ID

  5725L2900

Applicable component levels