IBM Support

The IBM Content Navigator server might hang if the IBM Content Navigator data source connection pool setting is too small

Troubleshooting


Problem

In a WebSphere Application Server environment, the default maximum data source connection pool setting is 10. As load on the server increases, users might see poor performance, and if the load is too large, the application server can hang.

Symptom

When the Content Navigator server hangs, the following errors might occur because the java threads cannot get a connection as resources have been exhausted:

  • WSVR0605W in the WebSphere application server log.
  • com/ibm/ejs/j2c/FreePool.createOrWaitForConnection(FreePool.java:1368) in the java core.

Here are more complete versions of the error messages you might see.

Sample error that might be seen in the application server log

[3/24/24 19:07:33:258 PDT] 000000a9 ThreadMonitor W WSVR0605W: Thread "WebContainer : 31" (0000061d) has been active for 751117 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.

at com.ibm.ecm.configuration.ConfigurationObject.load(ConfigurationObject.java:299)
at com.ibm.ecm.configuration.ConfigurationObject.(ConfigurationObject.java:107)
at com.ibm.ecm.configuration.UserConfig.(UserConfig.java:64)
at sun.reflect.GeneratedConstructorAccessor152.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:57)
at java.lang.reflect.Constructor.newInstance(Constructor.java:437)
at com.ibm.ecm.configuration.GlobalConfig._getConfiguration(GlobalConfig.java:886

Sample error that might be seen in the java core

In the javacore, all hung threads are waiting for this thread to get a connection from the free pool:
----------
3XMTHREADINFO "WebContainer : 17" J9VMThread:0x0000000002584800, omrthread_t:0x00007F9A7C067E78, java/lang/Thread:0x00000007F999CA60, state:CW, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0x136, isDaemon:true)
3XMJAVALTHRCCL com/ibm/ws/classloader/CompoundClassLoader(0x0000000682969FA8)
3XMTHREADINFO1 (native thread ID:0x3BEE, native priority:0x5, native policy:UNKNOWN, vmstate:CW, vm thread flags:0x00080181)
3XMTHREADINFO2 (native stack address range from:0x00007F9AD5AEC000, to:0x00007F9AD5B2C000, size:0x40000)
3XMCPUTIME CPU usage total: 0.300787064 secs, current category="Application"
3MTHREADBLOCK Waiting on: java/lang/Integer@0x000000068F087748 Owned by:
3XMHEAPALLOC Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/Object.wait(Native Method)
4XESTACKTRACE at java/lang/Object.wait(Object.java:216(Compiled Code))
4XESTACKTRACE at com/ibm/ejs/j2c/FreePool.queueRequest(FreePool.java:444)
4XESTACKTRACE at com/ibm/ejs/j2c/FreePool.createOrWaitForConnection(FreePool.java:1368)
4XESTACKTRACE at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:3885(Compiled Code))
4XESTACKTRACE at com/ibm/ejs/j2c/PoolManager.reserve(PoolManager.java:3133(Compiled Code))
4XESTACKTRACE at com/ibm/ejs/j2c/ConnectionManager.allocateMCWrapper(ConnectionManager.java:1556(Compiled Code))
4XESTACKTRACE at com/ibm/ejs/j2c/ConnectionManager.allocateConnection(ConnectionManager.java:1035(Compiled Code))
4XESTACKTRACE at com/ibm/ws/rsadapter/jdbc/WSJdbcDataSource.getConnection(WSJdbcDataSource.java:646(Compiled Code))

Cause

The load on the system is outpacing the availability of data source connections and java threads are hanging waiting on the availability of a free connection.

Resolving The Problem

The issue can be resolved in two ways.

Option 1

Increase the size of the data source connection pool and reduce the connection timeout value. This topic in the IBM Content Navigator documentation provides the procedure for making the change: https://www.ibm.com/docs/en/content-navigator/3.0.15?topic=ptcn-increasing-jdbc-connection-pool-sizes-websphere-application-server.

To increase the JDBC connection pool size:

  1. In WebSphere Application Server, navigate to Resources > JDBC > Data Sources.
  2. Select the data source to modify; for instance ECMClientDS and ECMTaskManagerDS
  3. Click Connection pool properties.
  4. Adjust the values of Information required Connection timeout and Information required Maximum connections
  5. Change the default number of connections for the Maximum Connections setting. Change the setting from 10 to a higher number, such as 100.
  6. Click Apply.

Over time, gradually adjust the maximum value to allow for more connections if necessary. You can increase the number of connections by 100. If performance begins to decrease, increase the number again by another 100 connections

Option 2

Adjust the Content Navigator configuration Get Property method timeout on the application server using a Java Virtual Machine (JVM) argument.

To set the Get Property method timeout

  1. In the WebSphere Application Server console, navigate to Process Definition > Java Virtual Machine
  2. In the Generic JVM arguments field, add the following argument. The value is in milliseconds.

    -Dcom.ibm.ecm.icn.system.config.readTimeout=<value>

    For example, the following statement sets a timeout of 5 seconds:

    -Dcom.ibm.ecm.icn.system.config.readTimeout=5000

Document Location

Worldwide

[{"Type":"MASTER","Line of Business":{"code":"LOB18","label":"Miscellaneous LOB"},"Business Unit":{"code":"BU056","label":"Miscellaneous"},"Product":{"code":"SSEUEX","label":"IBM Content Navigator"},"ARM Category":[{"code":"a8m3p000000LRzBAAW","label":"ICN-\u003EPerformance"}],"ARM Case Number":"","Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"All Versions"}]

Document Information

Modified date:
01 April 2024

UID

ibm17145502

Page Feedback