Fixes are available
7.0.0.37: WebSphere Application Server V7.0 Fix Pack 37
8.5.5.5: WebSphere Application Server V8.5.5 Fix Pack 5
8.5.5.6: WebSphere Application Server V8.5.5 Fix Pack 6
8.0.0.11: WebSphere Application Server V8.0 Fix Pack 11
8.5.5.7: WebSphere Application Server V8.5.5 Fix Pack 7
7.0.0.39: WebSphere Application Server V7.0 Fix Pack 39
8.5.5.8: WebSphere Application Server V8.5.5 Fix Pack 8
8.0.0.12: WebSphere Application Server V8.0 Fix Pack 12
8.5.5.9: WebSphere Application Server V8.5.5 Fix Pack 9
7.0.0.41: WebSphere Application Server V7.0 Fix Pack 41
8.5.5.10: WebSphere Application Server V8.5.5 Fix Pack 10
8.5.5.11: WebSphere Application Server V8.5.5 Fix Pack 11
8.0.0.13: WebSphere Application Server V8.0 Fix Pack 13
7.0.0.43: WebSphere Application Server V7.0 Fix Pack 43
8.5.5.12: WebSphere Application Server V8.5.5 Fix Pack 12
8.0.0.14: WebSphere Application Server V8.0 Fix Pack 14
8.5.5.13: WebSphere Application Server V8.5.5 Fix Pack 13
7.0.0.45: WebSphere Application Server V7.0 Fix Pack 45
8.0.0.15: WebSphere Application Server V8.0 Fix Pack 15
7.0.0.45: Java SDK 1.6 SR16 FP60 Cumulative Fix for WebSphere Application Server
7.0.0.37: Java SDK 1.6 SR16 FP3 Cumulative Fix for WebSphere Application Server
7.0.0.39: Java SDK 1.6 SR16 FP7 Cumulative Fix for WebSphere Application Server
7.0.0.41: Java SDK 1.6 SR16 FP20 Cumulative Fix for WebSphere Application Server
7.0.0.43: Java SDK 1.6 SR16 FP41 Cumulative Fix for WebSphere Application Server
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.15: WebSphere Application Server V8.5.5 Fix Pack 15
8.5.5.14: WebSphere Application Server V8.5.5 Fix Pack 14
8.5.5.17: WebSphere Application Server V8.5.5 Fix Pack 17
8.5.5.20: WebSphere Application Server V8.5.5.20
8.5.5.18: WebSphere Application Server V8.5.5 Fix Pack 18
8.5.5.19: WebSphere Application Server V8.5.5 Fix Pack 19
8.5.5.16: WebSphere Application Server V8.5.5 Fix Pack 16
8.5.5.21: WebSphere Application Server V8.5.5.21
APAR status
Closed as program error.
Error description
High CPU utilization may be seen when a change in security configuration is made on a deployment manager that manages a large number of application servers and the configuration change is synchronized with the nodes. Thread dumps may show a "Non-deferrable Alarm" thread in a state like the following: "Non-deferrable Alarm : 1" daemon prio=3 tid=0x0000000101abb800 nid=0x43 runnable [0xffffffff3bcfd000] java.lang.Thread.State: RUNNABLE at com.ibm.crypto.provider.lc.a(Unknown Source) at com.ibm.crypto.provider.lc.decrypt(Unknown Source) at com.ibm.crypto.provider.xb.decrypt(Unknown Source) at com.ibm.crypto.provider.tb.b(Unknown Source) at com.ibm.crypto.provider.RC2.a(Unknown Source) at com.ibm.crypto.provider.RC2.engineDoFinal(Unknown Source) at com.ibm.crypto.provider.RC2.engineDoFinal(Unknown Source) at com.ibm.crypto.provider.PBEWithSHAAnd40BitRC2.engineDoFinal(Unkn own Source) at javax.crypto.Cipher.doFinal(Unknown Source) at com.ibm.security.pkcs12.BasicPFX.getBags(BasicPFX.java:2094) at com.ibm.security.pkcs12.BasicPFX.getSafeBags(BasicPFX.java:1654) at com.ibm.security.pkcs12.BasicPFX.getSafeBagsByFriendlyName(Basic PFX.java:1746) at com.ibm.security.pkcs12.BasicPFX.getPrivateKeysByFriendlyName(Ba sicPFX.java:1053) at com.ibm.security.pkcs12.PFX.getPrivateKeysByFriendlyName(PFX.jav a:630) at com.ibm.crypto.provider.PKCS12KeyStore.engineLoad(Unknown Source) - locked <0xffffffff547df830> (a java.util.Hashtable) at java.security.KeyStore.load(KeyStore.java:1185) at com.ibm.ws.ssl.config.WSKeyStore$1.run(WSKeyStore.java:734) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessCon troller.java:118) at com.ibm.ws.ssl.config.WSKeyStore.do_getKeyStore(WSKeyStore.java: 632) - locked <0xffffffff542d9160> (a com.ibm.ws.ssl.config.WSKeyStore) at com.ibm.ws.ssl.config.WSKeyStore.getKeyStore(WSKeyStore.java:962 ) at com.ibm.ws.security.config.AuthMechanismConfigImpl.do_initialize RSAProperties(AuthMechanismConfigImpl.java:608) - locked <0xffffffff51160338> (a com.ibm.ws.security.config.AuthMechanismConfigImpl) at com.ibm.ws.security.config.AuthMechanismConfigImpl.reinitializeR SAProperties(AuthMechanismConfigImpl.java:825) at com.ibm.ws.security.core.distSecurityComponentImpl.configChanged (distSecurityComponentImpl.java:1904) - locked <0xffffffff5058a7b8> (a java.lang.Object) at com.ibm.ws.management.component.ConfigChangeHandler$ConfigChange EventDispatcher.run(ConfigChangeHandler.java:344) at com.ibm.ws.management.component.ConfigChangeHandler.alarm(Config ChangeHandler.java:158) at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656) "Non-deferrable Alarm : 1" daemon prio=3 tid=0x0000000101abb800 nid=0x43 runnable [0xffffffff3bcfd000] java.lang.Thread.State: RUNNABLE at java.math.BigInteger.oddModPow(BigInteger.java:1821) at java.math.BigInteger.modPow(BigInteger.java:1571) at com.ibm.crypto.provider.c.a(Unknown Source) at com.ibm.crypto.provider.c.a(Unknown Source) at com.ibm.crypto.provider.uc.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1128) at java.security.Signature.sign(Signature.java:522) at com.ibm.crypto.provider.PKCS12KeyStore.a(Unknown Source) at com.ibm.crypto.provider.PKCS12KeyStore.engineLoad(Unknown Source) - locked <0xffffffff547df830> (a java.util.Hashtable) at java.security.KeyStore.load(KeyStore.java:1185) at com.ibm.ws.ssl.config.WSKeyStore$1.run(WSKeyStore.java:734) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessCon troller.java:118) at com.ibm.ws.ssl.config.WSKeyStore.do_getKeyStore(WSKeyStore.java: 632) - locked <0xffffffff542d9160> (a com.ibm.ws.ssl.config.WSKeyStore) at com.ibm.ws.ssl.config.WSKeyStore.getKeyStore(WSKeyStore.java:962 ) at com.ibm.ws.security.config.AuthMechanismConfigImpl.do_initialize RSAProperties(AuthMechanismConfigImpl.java:608) - locked <0xffffffff51160338> (a com.ibm.ws.security.config.AuthMechanismConfigImpl) at com.ibm.ws.security.config.AuthMechanismConfigImpl.reinitializeR SAProperties(AuthMechanismConfigImpl.java:825) at com.ibm.ws.security.core.distSecurityComponentImpl.configChanged (distSecurityComponentImpl.java:1904) - locked <0xffffffff5058a7b8> (a java.lang.Object) at com.ibm.ws.management.component.ConfigChangeHandler$ConfigChange EventDispatcher.run(ConfigChangeHandler.java:344) at com.ibm.ws.management.component.ConfigChangeHandler.alarm(Config ChangeHandler.java:158) at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656) The deployment manager SystemOut.log may show messages like these: UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 2" (00000010) to the alarm thread pool has been delayed for 48950 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows: at com.ibm.security.util.calendar.BaseCalendar$Date.<init>(BaseCale ndar.java:147) at com.ibm.security.util.calendar.Gregorian$Date.<init>(Gregorian.j ava:27) at com.ibm.security.util.calendar.Gregorian.newCalendarDate(Gregori an.java:51) at com.ibm.security.util.DerInputStream.getTime(DerInputStream.java :915) at com.ibm.security.util.DerInputStream.getUTCTime(DerInputStream.j ava:637) at com.ibm.security.x509.CertificateValidity.construct(CertificateV alidity.java:124) at com.ibm.security.x509.CertificateValidity.<init>(CertificateVali dity.java:201) at com.ibm.security.x509.X509CertInfo.parse(X509CertInfo.java:949) at com.ibm.security.x509.X509CertInfo.<init>(X509CertInfo.java:236) at com.ibm.security.x509.X509CertInfo.<init>(X509CertInfo.java:222) at com.ibm.security.x509.X509CertImpl.parse(X509CertImpl.java:2293) at com.ibm.security.x509.X509CertImpl.<init>(X509CertImpl.java:231) at com.ibm.security.x509.X509CertImpl.<init>(X509CertImpl.java:217) at com.ibm.security.pkcs12.CertBag.decode(CertBag.java:599) at com.ibm.security.pkcsutil.PKCSDerObject.decode(PKCSDerObject.jav a:252) at com.ibm.security.pkcs12.CertBag.<init>(CertBag.java:76) at com.ibm.security.pkcs12.BasicPFX.getCertificates(BasicPFX.java:1 469) at com.ibm.security.pkcs12.PFX.getCertificates(PFX.java:549) at com.ibm.crypto.provider.PKCS12KeyStore.engineLoad(Unknown Source) at java.security.KeyStore.load(KeyStore.java:1185) at com.ibm.ws.ssl.config.WSKeyStore$1.run(WSKeyStore.java:734) at com.ibm.ws.security.util.AccessController.doPrivileged(AccessCon troller.java:118) at com.ibm.ws.ssl.config.WSKeyStore.do_getKeyStore(WSKeyStore.java: 632) at com.ibm.ws.ssl.config.WSKeyStore.getKeyStore(WSKeyStore.java:962 ) at com.ibm.ws.security.config.AuthMechanismConfigImpl.do_initialize RSAProperties(AuthMechanismConfigImpl.java:608) at com.ibm.ws.security.config.AuthMechanismConfigImpl.reinitializeR SAProperties(AuthMechanismConfigImpl.java:825) at com.ibm.ws.security.core.distSecurityComponentImpl.configChanged (distSecurityComponentImpl.java:1904) at com.ibm.ws.management.component.ConfigChangeHandler$ConfigChange EventDispatcher.run(ConfigChangeHandler.java:344) at com.ibm.ws.management.component.ConfigChangeHandler.alarm(Config ChangeHandler.java:158) at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656). UTLS0008W: The return of alarm thread "Non-deferrable Alarm : 0" (0000000e) to the alarm thread pool has been delayed for 49793 milliseconds. This may be preventing normal alarm function within the application server. The alarm listener stack trace is as follows: at com.ibm.ws.security.core.distSecurityComponentImpl.configChanged (distSecurityComponentImpl.java:1877) at com.ibm.ws.management.component.ConfigChangeHandler$ConfigChange EventDispatcher.run(ConfigChangeHandler.java:344) at com.ibm.ws.management.component.ConfigChangeHandler.alarm(Config ChangeHandler.java:158) at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133) at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1656).
Local fix
Problem summary
**************************************************************** * USERS AFFECTED: All users of IBM WebSphere Application * * Server * **************************************************************** * PROBLEM DESCRIPTION: High CPU usage when deployment * * manager runs with many nodes on a * * server with certificate expiration * * monitor enabled. * **************************************************************** * RECOMMENDATION: * **************************************************************** When Certificate Expiration Monitor updates certificate on a deployment manager and try to propagate the change to many nodes, SSL-related reinitialization is triggered on each node. Among reinitialization, RSA token initialization which is only used for flexible management (AdminAgent and JobManager) takes up a lot of CPU utilization.
Problem conclusion
This apar introduces a new custom property: com.ibm.websphere.security.initializeRSAProperties If this property is set to "false", the application server will not perform re-initialization of RSA token related SSL properties. Default value is "true" Before configuring this property to "false", make sure Job Manager or Admin Agent is not used in the environment. These features do require RSA token and this apar code should not be used. If high CPU utilization is observed in Job Manager / Admin Agent environment after Certificate Expieraiton Monitor run, nodes may need to be distributed to multiple servers to reduce the CPU load on one server. The fix for this APAR is currently targeted for inclusion in fix pack 7.0.0.37, 8.0.0.11 and 8.5.5.5. Please refer to the Recommended Updates page for delivery information: http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980
Temporary fix
Comments
APAR Information
APAR number
PI29000
Reported component name
WEBS APP SERV N
Reported component ID
5724H8800
Reported release
700
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt / Xsystem
Submitted date
2014-11-04
Closed date
2015-01-06
Last modified date
2015-01-06
APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:
Fix information
Fixed component name
WEBS APP SERV N
Fixed component ID
5724H8800
Applicable component levels
R700 PSY
UP
R800 PSY
UP
R850 PSY
UP
[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]
Document Information
Modified date:
27 April 2022