Technical Blog Post
Abstract
High CPU Usage caused by Jetty btpool threads.
Body
High CPU usage by Jetty bounded thread pool (btpool) thread:
There might be several reasons for high CPU utilization. However in this blog am writing high CPU utilization caused by jetty btpool threads. The issue is caused by underlying jetty class - org.mortbay.jetty.nio.SelectChannelConnector.
When the user finds high CPU utilization on the server verify if that is caused by btpool threads. If the symptoms are matching as below then apply the fix recommended below.
Identifying if this is caused by btpool threads.
1. Run "top -H" on the operating system. Verify which threads are consuming more CPUs. If this issue is caused by the btpool thread then you would see btpool threads.
2. Take few thread dumps. The thread dump can be taken either from command line or from dashboard Operations > System > Performance > JVM Monitor page. Refer the below documentation link on taking thread dumps.
Once the thread dump is taken navigate to <SBI_INSTALL_DIR>/noapp/bin directory. Then open the thread dump in the text editor. Search for text "CPU usage". Verify the CPU usage by various threads and see if the high CPU usage is corresponding to the btpool threads. The thread dump should show something like below for the jetty btpool threads.
3XMTHREADINFO "btpool0-2 - Acceptor0 SelectChannelConnector@pedisfg02:5000" J9VMThread:0x0000010027DDC600, j9thread_t:0x0000010024C35DA0, java/lang/Thread:0x0A00000000B20B80, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x73, isDaemon:false)
3XMTHREADINFO1 (native thread ID:0x2B501BB, native priority:0x5, native policy:UNKNOWN)
3XMCPUTIME CPU usage total: 31604.141012000 secs, user: 17275.460932000 secs, system: 14328.680080000 secs
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at sun/nio/ch/PollArrayWrapper.poll0(Native Method)
4XESTACKTRACE at sun/nio/ch/PollArrayWrapper.poll(PollArrayWrapper.java:147(Compiled Code))
4XESTACKTRACE at sun/nio/ch/PollSelectorImpl.doSelect(PollSelectorImpl.java:90(Compiled Code))
4XESTACKTRACE at sun/nio/ch/SelectorImpl.lockAndDoSelect(SelectorImpl.java:99(Compiled Code))
5XESTACKTRACE (entered lock: sun/nio/ch/PollSelectorImpl@0x0A00000039F7B590, entry count: 1)
5XESTACKTRACE (entered lock: java/util/Collections$UnmodifiableSet@0x0A00000039F756C8, entry count: 1)
5XESTACKTRACE (entered lock: sun/nio/ch/Util$2@0x0A00000039F756E0, entry count: 1)
4XESTACKTRACE at sun/nio/ch/SelectorImpl.select(SelectorImpl.java:110(Compiled Code))
4XESTACKTRACE at org/mortbay/io/nio/SelectorManager$SelectSet.doSelect(SelectorManager.java:498(Compiled Code))
4XESTACKTRACE at org/mortbay/io/nio/SelectorManager.doSelect(SelectorManager.java:192(Compiled Code))
4XESTACKTRACE at org/mortbay/jetty/nio/SelectChannelConnector.accept(SelectChannelConnector.java:124(Compiled Code))
4XESTACKTRACE at org/mortbay/jetty/AbstractConnector$Acceptor.run(AbstractConnector.java:708(Compiled Code))
4XESTACKTRACE at org/mortbay/thread/BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:451)
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK _event_wait+0x344 (0x09000000004EF068 [libpthreads.a+0x17068])
4XENATIVESTACK _cond_wait_local+0x4dc (0x09000000004FD1A0 [libpthreads.a+0x251a0])
4XENATIVESTACK _cond_wait+0xc8 (0x09000000004FD78C [libpthreads.a+0x2578c])
4XENATIVESTACK pthread_cond_wait+0x19c (0x09000000004FE3E0 [libpthreads.a+0x263e0])
4XENATIVESTACK (0x0900000006C1A2F4 [libj9thr26.so+0x62f4])
4XENATIVESTACK (0x0900000006C19F40 [libj9thr26.so+0x5f40])
4XENATIVESTACK (0x0900000007861C58 [libj9vm26.so+0xfc58])
4XENATIVESTACK (0x09000000081CE588 [libj9jit26.so+0x7e6588])
4XENATIVESTACK (0x09000000078585E4 [libj9vm26.so+0x65e4])
4XENATIVESTACK (0x0900000007915CE0 [libj9prt26.so+0x2ce0])
4XENATIVESTACK (0x0900000007858454 [libj9vm26.so+0x6454])
4XENATIVESTACK (0x0900000006C16AF4 [libj9thr26.so+0x2af4])
4XENATIVESTACK _pthread_body+0xf0 (0x09000000004DBE14 [libpthreads.a+0x3e14])
NULL
If you see the same symptoms as above then follow the steps below to resolve this issue. If you are running on Sterling B2B Integrator 5.2.4.2 then apply 5020402_5 or higher. If you are running on Sterling B2B Integrator 5.2.5.0 then apply 5020500_1 or higher.
Once the needed fixpack installed the user can activate the fix by following the steps below. The fix is disabled by default. To activate,
1. Stop IBM Sterling B2B Integrator.
2. Add a property usingNIOConnector=false in noapp.properties.in file and run setupfiles.sh
or
Add noapp.usingNIOConnector=false to customer_overrides.properties.
3. Start IBM Sterling B2B Integrator.
Please post your queries if there are any. Please open a PMR with IBM support team if this does not resolve the issue or if there are queries further.
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SS3JSW","label":"IBM Sterling B2B Integrator"},"Component":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"","Edition":"","Line of Business":{"code":"LOB59","label":"Sustainability Software"}}]
UID
ibm11121583