IBM Support

Troubleshooting connection pooling (J2C) problems in WebSphere Application Server

Troubleshooting


Problem

Troubleshooting for connection pooling problems in IBM® WebSphere® Application Server. This should help address common issues with this component before calling IBM support and save you time. Note: The troubleshooting steps in this document are applicable to JDBC connection problems using a standard data source in WebSphere Application Server. This document is not applicable to Version 4 data sources. You can also use this document for troubleshooting JMS connection problems and enterprise information system (EIS) connection problems.

Resolving The Problem

Tab navigation

Steps to help resolve connection pooling problems

  1. Are you having a problem using a data source to establish a JDBC connection to a database? Most exceptions that occur when a data source is used to connect to a database will have the com.ibm.ws.rsadapter package in the stack trace of the exception. This includes java.sql.SQLExceptions and WebSphere Application Server messages that begin with DSRA.
  2. Are you having a problem using a JMS connection factory to establish a JMS connection to WebSphere MQ or another messaging system? Most exceptions that occur when a connection factory is used to connect to a messaging system will have the com.ibm.ejs.jms package in the stack trace of the exception. This includes javax.jms.JMSExceptions and WebSphere Application Server messages that begin with WMSG.
  3. Are you seeing J2CA0045E errors with ConnectionWaitTimeoutExceptions or slow performance in getting a connection?
  4. Are you seeing StaleConnectionExceptions or having problems recovering from invalid, or stale, connections in the pool?
  5. Go to the Miscellaneous connection pooling problems section. If your problem is not covered in that section or addressed elsewhere in this document, review the WebSphere Application Server V8.0, V8.5.5, or V9.0 product documentation, or the WebSphere Application Server Support site for additional information that might help you to resolve the problem, or continue to the MustGather: Connection pooling problems for WebSphere Application Server for database connection and connection pooling problems.

Troubleshooting JDBC connection problems
  1. This section will help you to troubleshoot problems that occur when using a data source defined in WebSphere Application Server to establish a JDBC connection to a database. JDBC connection problems usually occur at runtime when an application attempts to connect to the database and perform SQL queries. This section does not include ConnectionWaitTimeoutExceptions or StaleConnectionExceptions, which are covered in subsequent sections.

    To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. When diagnosing a database connection problem, the first step is to use the Test Connection button for the data source in the administrative console to test the connection to the database. You can find the Test Connection button in the data source configuration panel.
     
  2. Was the attempt to connect to the database using the Test Connection button successful?
    • Yes, continue to question 3.
    • No, continue to question 4.
       
  3. If you can use the Test Connection button to successfully connect to the database, but a failure occurs when your application tries to get a connection, the problem is likely caused by a failed JNDI lookup. Check the SystemOut.log to see if a NameNotFoundException occurs when the application tries to use the data source. Does a NameNotFoundException occur?
    • Yes, the root cause of the problem is that the JNDI lookup of the data source by the application fails. Check the data source JNDI name and ensure that it is bound to the JNDI namespace when the application server starts. If the JNDI name is jdbc/ds, you should see this entry in the SystemOut.log:
      WSVR0049I: Binding ds as jdbc/ds
      Also, check the application code and ensure that it is looking up the correct JNDI name. If you are doing an indirect JNDI lookup (i.e. java:comp/env/jdbc/ds), ensure that the binding is correct in the data source resource reference. The resource reference is configured in the web, EJB, or application client deployment descriptor. Checking for these things can help you to resolve many JNDI problems. However, if you can't determine the cause of the problem, continue to the Troubleshooting JNDI and naming problems.
       
    • No, continue to question 8.
       
  4. The error message shown in the administrative console when the Test Connection button fails is important to understanding the cause of the problem. You can also check the SystemOut.log to get more information about the error and see the stack trace.

    Does the error message and the accompanying SQLException indicate a problem with the user ID and password that are used to connect to the database? The actual error and exception will differ depending on the JDBC driver and database that you're using.
    • Yes, the root cause of the problem is either that no userid and password are passed to the database, or the userid or password is not correct. Ensure that a J2C authentication alias, containing the correct userid and password, is specified on the data source. Also check with your DBA to make sure that the userid and password that you are using are correct for connecting to the database.
       
    • No, continue to question 5.
       
  5. If you are using a Type 2 driver or Type 3 JDBC driver, does a java.lang.UnsatisfiedLinkError occur when you use the Test Connection button? A java.lang.UnsatisfiedLinkError occurs when the JVM is unable to load a native library that is needed by the JDBC driver.
    • Yes, the root cause of the problem is that the application server JVM is not properly configured to load the native libraries. For a DB2 database, see documents java.lang.UnsatisfiedLinkError: SQLConnect error connecting to DB2 type 2 datasource, Failure in loading native library db2jcct2, UnsatisfiedLinkError: ERRORCODE=-4472, and WebSphere database connection exception java.lang.UnsatisfiedLinkError. If you are trying to connect to an Oracle database, ensure that the ORACLE_HOME and the LIBPATH (on AIX®), LD_LIBRARY_PATH (on Sun Solaris or Linux®), or SHLIB_PATH (on HP-UX) JVM environment entries are set. The required JVM environment entries might be different for other databases.

      You should also make sure that the user that is being used to run the application server has the proper permissions to access the native libraries. Incorrect permissions are another common cause of the java.lang.UnsatisfiedLinkError.

      Finally, remember that 32-bit native libraries cannot be used with a 64-bit application server JVM and vice versa. Mismatches between the two can also cause the java.lang.UnsatisfiedLinkError.
       
    • No, continue to question 6.
       
  6. If you are testing a connection to a Microsoft® SQL Server database, does the following error occur?
    java.sql.SQLException: [IBM][SQLServer JDBC Driver][SQLServer]Could not find stored procedure 'master..xp_jdbc_open'.
  7. If the error that you are getting is not one of the common errors described in the previous steps, review the error message and SQLException that occur when using the Test Connection button. Check the properties that you have specified on the data source to ensure that they are correct. Refer to your database documentation or your DBA for more information about why a particular error or exception might have occurred. An example of a typical error message that might occur is:


    DSRA8040I: Failed to connect to the DataSource.  Encountered : java.lang.Exception:          java.sql.SQLException: Io exception: The Network Adapter could not establish the connectionDSRA0010E: SQL State = null, Error Code = 17,002
    If you still cannot resolve the problem, continue to question 8.
     
  8. Finally, if none of the previous troubleshooting steps helped to resolve the problem, try to connect to the database server using a stand-alone java program to determine if you are able to successfully connect to the database using the same properties that you have set in your data source outside of WebSphere Application Server. This will allow you to determine whether the problem is specific to WebSphere Application Server or if the problem occurs with the JDBC driver independently of WebSphere Application Server..
    • If a direct JDBC connection fails with the same error that occurs within WebSphere Application Server, the problem is not specific to your WebSphere Application Server environment. Work with your DBA to troubleshoot the problem.
       
    • If the direct JDBC connection is successful, continue to the MustGather: Connection pooling problems for WebSphere Application Server for database connection and connection pooling problems.

Troubleshooting JMS connection problems
  1. This section will help you to troubleshoot problems that occur when using a JMS connection factory to establish a connection to a messaging system such as WebSphere MQ. Only JMS connection problems are covered here.

    To troubleshoot this type of problem, you should have access to the SystemOut.log for your application server. When troubleshooting a JMS connection problem to MQ, it is useful to determine the MQ reason code associated with the JMSException. In the SystemOut.log, find the JMSException and review the stack trace. You should see one or more linked exceptions. Find the last linked exception. You should see a line like this:
    ---- Begin backtrace for Nested Throwables
    com.ibm.mq.MQException: MQJE001: Completion Code 2, Reason 2009

    The four-digit number is the reason code. You can review this Messages document to find out more about the reason code that you see. The reason code helps you to understand the root cause of the problem.
     
  2. Are you receiving a JMSException with MQ reason code 2009?
  3. Are you receiving a JMSException with MQ reason code 2019?
    • Yes, the root cause of the problem could be that the connection pool Purge Policy is not set correctly. Review technote JMS connections fail with Reason Code 2019 to resolve the problem.
       
    • No, continue to question 4.
       
  4. Are you receiving the following exception?


    J2CA0020E: The Connection Pool Manager could not allocate a Managed Connection: java.lang.IllegalStateException: Internal Error: cannot find the PoolManager Reference
  5. Are either of the following authentication errors occurring when trying to establish a JMS connection?


    MSGS0508E: The JMS Server security service was unable to authenticate userid: myuser

    javax.jms.JMSSecurityException: MQJMS2013: invalid security authentication supplied for MQQueueManager
    • Yes, the root cause of the problem is either that no userid and password are passed to the queue manager, or the userid or password is not correct. Ensure that a J2C authentication alias is specified on your JMS connection factory.
    • No, continue to question 6.
       
  6. Is the problem that you observe (using a tool such as netstat) more TCP/IP connections than you expect between your application server process and an MQ queue manager?
  7. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather: Connection pooling problems for WebSphere Application Server for connection pooling problems.


Troubleshooting connection wait issues, connection leaks, and performance problems
  1. This section will help you to troubleshoot connection wait issues, including those that are caused by application connection leaks, and general performance issues that are related to connection pooling. These types of problems could occur for any type of backend system, including databases, messaging systems, and enterprise information systems.

    To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. It is also preferable for you to have access to the source code for your application(s). Additionally, obtaining javacores (also known as thread dumps) may help you to resolve the problem.




The following documents can be used to diagnose the J2CA0045E error:
  • How to troubleshoot J2CA0045E connection pooling problems.
  • Webcast Reply: Understanding and Resolving ConnectionWaitTimeout Exceptions in WAS
  • What does J2CA0045E mean for WebSphere Application Server?
  • Using Connection information in WebSphere trace files to troubleshoot J2CA0045E and J2CA0020E or connection wait time-out problems
  • Application hangs with J2CA0045E errors and with J2CA0086W warning messages

  • 1. Are you experiencing a connection wait problem? Symptoms might include ConnectionWaitTimeoutExceptions in the SystemOut.log file, slower performance or delays when the application tries to get a connection, or threads hanging while waiting for a free connection (this can be seen in a javacore). An example stack trace for a thread that is waiting for a free connection is:

    "Servlet.Engine.Transports : 2833" (TID:0x31680590, sys_thread_t:0x78C96AA0, state:CW, native ID:0x86C7) prio=5
        at java.lang.Object.wait(Native Method)
        at com.ibm.ejs.j2c.poolmanager.FreePool.queueRequest(FreePool.java(Compiled Code))
        at com.ibm.ejs.j2c.poolmanager.FreePool.createOrWaitForConnection(FreePool.java(Compiled Code))
        at com.ibm.ejs.j2c.poolmanager.PoolManager.reserve(PoolManager.java(Compiled Code))
    ...
    • Yes, continue to question 4.
       
    • No, continue to question 9.
       
    2. This situation would occur when the connection pool is at its maximum size (defined by the Maximum Connections property on the connection pool), all of the connections are in use, and the application requests more connections. The application must wait for a free connection to become available.

    Does the problem eventually clear up or does it persist until you restart the application server?
    • If the problem eventually clears up, the root cause of the problem is likely that Maximum Connections is not set high enough for the amount of load on the application. You should increase Maximum Connections for the connection pool. You should conduct thorough load testing to find the optimal value for Maximum Connections. You can enable PMI and monitor the connection pool counters in the Tivoli Performance Viewer to help you tune this.
       
    • If the problem does not clear up until you restart the application server, continue to question 5.
       
    3. Is the application calling close() on every connection object that it obtains from a WebSphere Application Server connection pool? When an application gets a connection from the pool, it is considered "in use" until the application calls close() on the connection, which then returns the connection to the free pool. If the application does not call close(), the connection is leaked and never returns to the free pool. Eventually, the pool might become filled with leaked connections, causing connection wait problems.

    To determine if an application is leaking connections, you should review the application source code. Note that the application should close connections in a finally block to ensure that the connections are closed even in error conditions. In addition, you can use WebSphere Application Server tracing to identify connection leaks. The trace output can show you that a connection leak exists, and it can also show exactly what application code is leaking connections. Finally, the PMI connection pool counters might also be useful in determining whether or not there is a connection leak.
    • If the application is failing to call close() on all connection objects, that is the root cause of the problem. You should fix the application code to resolve the problem.
       
    • If you determine that the application server is not leaking connections, continue to question 6.
       
    4. Connection wait problems and slow application performance when using connection pooling can occur when the web container is not configured properly. If the web container thread pool size is set too high relative to the Maximum Connections setting for the connection pool, resource contention for the available connections could occur. It is also strongly recommended not to check the isGrowable checkbox for the web container thread pool.
    Is the web container thread pool size set too high or is the isGrowable checkbox checked in your configuration?
    • Yes, then this configuration is likely the root cause of the problem. Correct the issue and then test to see if the problem is resolved.
       
    • No, continue to question 7.
       
    5. Other issues might cause connection wait problems. Applications should follow the "get/use/close" pattern and close all connections immediately after finishing using them. If an application takes too long to close connections, or caches connections, the connections will be in use longer, which could lead to connection wait issues. Even if the "get/use/close" pattern is followed, connections will not return to the free pool until the transaction in which the connection is obtained is committed. If the transaction remains active for a long time after a connection is closed, connection wait issues could occur. This frequently occurs in servlets when a shareable connection is obtained in a local transaction containment (LTC). More details about this scenario and solutions for it are documented in this Default behavior of managed connections in WebSphere Application Server article.

    Is your application following the "get/use/close" pattern and not caching connections, without long periods of time elapsing before the transaction is committed?
    • Yes, continue to question 8.
       
    • No, then the root cause of the problem is that connections are "in use" for too long before returning to the free pool. Correct the issue and then test to see if the problem is resolved.
       
    6. Are there any network or database issues at the time that the problem occurs? Connection wait problems and performance problems might occur when network traffic is slow or when the database server is busy or performing slowly.
    • Yes, then the root cause of the problem is that connections are "in use" for long periods of time due to the network or database slowdown. Correct the problem with the network or the database server.
       
    • No, continue to question 9.
       
    7. If you are seeing slow performance related to connection pooling without a connection wait issue, check the SystemOut.log. Are WSVR0605W warnings appearing in the SystemOut.log file?
    • Yes, this means that there is a thread that has been running as long as the threshold defined in the hang detection policy. Generate javacores to analyze why the thread is hung. This should reveal the root cause of the problem.
       
    • No, continue to question 10.
       
    8. If none of the conditions described in this section are occurring, but the application is performing poorly or the application server appears to be unresponsive or hung, then use the appropriate MustGather document for an application server hang to analyze the threads of the application server to determine the cause of the problem.


    9. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to theMustGather: Connection pooling problems for WebSphere Application Server for database connection and connection pooling problems.


Troubleshooting stale connection problems
  1. This section will help you to troubleshoot problems with stale or invalid connections in the connection pool. WebSphere Application Server can determine that a connection is stale based on the exception that is returned by the backend when trying to establish a connection. When there is a stale database connection, WebSphere Application Server issues a StaleConnectionException. For other backends, a fatal connection error occurs. As a result, WebSphere Application Server purges the connection pool based on the Purge Policy setting and applications are able to recover from the problem. The following two links have more information on this issue: WebSphere Application Server StaleConnectionExceptions and Demystifying the WebSphere StaleConnectionException.

    To troubleshoot this type of problem, you should have access to the administrative console and the SystemOut.log for your application server. It is also preferable for you to have access to the source code for your application(s).
     
  2. Is the backend system that you are connecting to a database, a messaging system, or an enterprise information system?
    • If it is a database, continue to question 3.
       
    • If it is a messaging system or an enterprise information system, continue to question 6.
       
  3. If you are seeing StaleConnectionExceptions in the SystemOut.log, do they occur for every connection attempt or do they occur intermittently or only after a certain time period has elapsed?
    • If it occurs for every connection attempt, the root cause of the problem is likely that the data source properties are misconfigured. Follow the steps in the Troubleshooting JDBC connections section.
       
    • If it occurs intermittently or only after a certain time period has elapsed, continue to question 4.
       
  4. Is the application handling the StaleConnectionException properly by catching the exception and then retrying the connection?
    • Yes, continue to question 5.
       
    • No, then the root cause of the problem is that the application is not catching the exception and retrying, so it cannot recover from the problem. Review the WebSphere Application Server StaleConnectionException document on how to handle the StaleConnectionException and implement this in your application code.
       
  5. A data source can be configured to pretest connections to ensure that they are valid before they are allocated to the application. Here is the document for pretest connections. Have you configured your data source to do this?
    • Yes, continue to question 6.
       
    • No, the problem can probably be resolved by configuring the data source to pretest connections before they are allocated to the application.
       
  6. Is the Purge Policy set to EntirePool for the connection pool? This means that when a StaleConnectionException occurs, every free connection in the pool will be purged, which makes it easier for the application to recover.
    • Yes, continue to question 7.
       
    • No, the problem symptoms can be alleviated by changing the Purge Policy to EntirePool.
       
  7. Is the Minimum Connections property in the connection pool set to 0? Setting it to 0 enables the pool maintenance thread to clean up all of the connections in the pool after they are unused for more than the Unused Timeout number of seconds. If Minimum Connections is set to a value greater than 0, WebSphere Application Server must keep at least that number of connections in the pool indefinitely. The longer that a connection remains in the pool, the more susceptible it is to becoming stale.
    • Yes, continue to question 8.
       
    • No, the problem symptoms can be alleviated by changing the value of Minimum Connections to 0.
       
  8. Does a firewall exist between the application server and the backend system that it is connecting to?
    • Yes, the root cause of the problem is likely that the firewall is timing out and dropping connections between the application server and the backend. To avoid this possibility, set the Unused Timeout to half the value of the timeout setting on the firewall. This way, WebSphere Application Server can clean up its unused connections before the firewall drops them.
       
    • No, continue to question 9.
       
  9. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather: Connection pooling problems for WebSphere Application Server for database connection and connection pooling problems. Also, see the following documents for additional information on the StaleConnection Exception:
Miscellaneous connection pooling problems
  1. Are you seeing J2CA0075W warnings in the SystemOut.log?
    J2CA0075W: An active transaction should be present while processing method allocateMCWrapper.
  2. Are you seeing J2CA0106E errors in the SystemOut.log?
    J2CA0106E: A 5.0 DataSource was attempted to be used in a WebModule that was not level 2.3
  3. Are you running WebSphere Application Server V7, V8.5.5 or V9 and seeing J2CA0294W warnings in the SystemOut.log?
    J2CA0294W: Deprecated usage of direct JNDI lookup of resource jdbc/ds. The following default values are used: [Resource-ref settings]
      res-auth: 1 (APPLICATION)
      res-isolation-level: 0 (TRANSACTION_NONE)
      res-sharing-scope: true (SHAREABLE)
      loginConfigurationName: null
      loginConfigProperties: null
      [Other attributes]

      res-resolution-control: 999 (undefined)
      isCMP1_x: false (not CMP1.x)
      isJMS: false (not JMS)
    • Yes, it is not necessarily a problem and it can be prevented by creating a resource reference and doing an indirect JNDI lookup of your data source or connection factory. Review this J2CA0294W: Deprecated usage of direct JNDI lookup of resource technote to resolve the problem.
       
    • No, continue to question 5.
       
  4. Are you seeing the following error in the SystemOut.log?
    J2CA0079E: Method getManagedConnection has detected an internal illegal state and is throwing an IllegalStateException. The exception is: java.lang.IllegalStateException: setManagedConnection: illegal state exception. State = STATE_INACTIVE
  5. Are you seeing J2CA0086W warnings in the SystemOut.log?
    J2CA0086W: Shareable connection MCWrapper id 686bbdf9 Managed connection WSRdbManagedConnectionImpl@6156bdf9 State:STATE_TRAN_WRAPPER_INUSE from resource jdbc/ds was used within a local transaction containment boundary.
  6. Are you seeing high memory usage or java.lang.OutOfMemoryErrors due to a high statement cache size?
  7. Are you seeing java.lang.ClassCastExceptions or other errors when using vendor specific, non-standard JDBC methods in your application?
    • Yes, the problem can probably be resolved by using the JDBC 4.0 Wrapper Pattern. This provides a standard way to access native connections and non-standard JDBC methods.
    • No, continue to question 9.
       
  8. Finally, if none of the previous troubleshooting steps helped to resolve the problem, continue to the MustGather: Connection pooling problems for WebSphere Application Server for database connection and connection pooling problems.

What to do next
If the preceding troubleshooting steps did not solve your problem, see the MustGather: Connection pooling problems for WebSphere Application Server for Connection Pooling problems to continue investigation.

Other related documents

[{"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Component":"DB Connections\/Connection Pooling","Platform":[{"code":"PF002","label":"AIX"},{"code":"PF010","label":"HP-UX"},{"code":"PF012","label":"IBM i"},{"code":"PF016","label":"Linux"},{"code":"PF027","label":"Solaris"},{"code":"PF033","label":"Windows"}],"Version":"9.0;8.5.5;8.0;7.0","Edition":"Base;Network Deployment","Line of Business":{"code":"LOB45","label":"Automation"}},{"Product":{"code":"SSNVBF","label":"Runtimes for Java Technology"},"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Component":"Java SDK","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"LOB36","label":"IBM Automation"}}]

Document Information

Modified date:
25 April 2021

UID

swg21247168

Page Feedback