Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
15 replies Latest Post - ‏2012-05-21T16:25:27Z by mojojojo
mojojojo
mojojojo
29 Posts
ACCEPTED ANSWER

Pinned topic many java.sql.SQLException (s)

‏2012-05-14T17:07:47Z |
Hi All,

Attached SystemOut.log contains many errors related to DB2.
You can ignore the errors "javax.jcr.RepositoryException: Failed to login to repository."

Scenario:
we recently upgraded our portal from v-6.1.0.1 to v-6.1.0.6. After that, we see many errors.

We had one more problem, which is tracked in https://www.ibm.com/developerworks/forums/message.jspa?messageID=14823057.
This problem is now solved, after the suggestion in that thread (i.e. adding servlet-filter).

I am yet to see the SQLException after this change (adding servlet-filter); as I am not sure, if this fix has solved the DB2 errors too. But I am creating this thread anyway.

Thank you for any suggestions on these db errors.
Updated on 2012-05-21T16:25:27Z at 2012-05-21T16:25:27Z by mojojojo
  • mojojojo
    mojojojo
    29 Posts
    ACCEPTED ANSWER

    Re: many java.sql.SQLException (s)

    ‏2012-05-14T17:09:38Z  in response to mojojojo
    Attached the log again.
    • mojojojo
      mojojojo
      29 Posts
      ACCEPTED ANSWER

      Re: many java.sql.SQLException (s)

      ‏2012-05-14T17:20:39Z  in response to mojojojo
      there was a problem with the size of attachment. truncated the log and attached again.

      Thanks
      • SystemAdmin
        SystemAdmin
        30895 Posts
        ACCEPTED ANSWER

        Re: many java.sql.SQLException (s)

        ‏2012-05-14T21:16:50Z  in response to mojojojo
        If you have a firewall between the application server and the database server, this looks to be the case where the firewall is dropping connections. I would start by checking the database connection pool configuration and set Minimum Connections to 0 and lower the Unused Timeout to half of the firewall timeout value. If you need more suggestions on troubleshooting connection pool issues, this Technote may be helpful:

        http://www-01.ibm.com/support/docview.wss?uid=swg21372849
        The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM.
        • mojojojo
          mojojojo
          29 Posts
          ACCEPTED ANSWER

          Re: many java.sql.SQLException (s)

          ‏2012-05-15T07:31:30Z  in response to SystemAdmin
          Thank you again Nguyen.

          I will check those settings.

          But, now i see some thread hanging errors and CPU Starvation detected as well.
          They are related to ESIProcessor, which seem to have been fixed.
          http://www-01.ibm.com/support/docview.wss?uid=swg1PM31557
          http://www-01.ibm.com/support/docview.wss?uid=swg1PK86245

          But these fixpacks would have been already included in our WPS-6.1.0.6 , WAS-6.1.0.43.
          Our upgrade was successful, without any errors. Did i miss any settings?
          • mojojojo
            mojojojo
            29 Posts
            ACCEPTED ANSWER

            Re: many java.sql.SQLException (s)

            ‏2012-05-15T07:36:03Z  in response to mojojojo
            attached logs for hung threads.
            • SystemAdmin
              SystemAdmin
              30895 Posts
              ACCEPTED ANSWER

              Re: many java.sql.SQLException (s)

              ‏2012-05-15T16:29:34Z  in response to mojojojo
              Yes, these threads are related to ESI processing. I looked up some old records, and the recommendation is that in addition to upgrade the WAS to the version that includes the fix, make sure the WAS SDK is also at the required level for the WAS version.

              Also, you may need to review the ESI configuration in the plugin-cfg.xml. I hope you'll find this document helpful.

              http://www-01.ibm.com/support/docview.wss?uid=swg27015501

              The postings on this site are my own and do not necessarily represent the positions, strategies, or opinions of IBM.
              • mojojojo
                mojojojo
                29 Posts
                ACCEPTED ANSWER

                Re: many java.sql.SQLException (s)

                ‏2012-05-15T16:54:48Z  in response to SystemAdmin
                Our WAS SDK is also at the pre-requisite level.
                Will check the ESI Configuration in plugin-cfg.xml.

                Thanks
      • SystemAdmin
        SystemAdmin
        30895 Posts
        ACCEPTED ANSWER

        Re: many java.sql.SQLException (s)

        ‏2012-05-16T14:02:41Z  in response to mojojojo
        This systemout.log file suggests that the hang is caused during the incomplete wcm syndication during the previous crash. The threads are timing out on the database connection.

        Time-outs during syndication are often caused by the failure of large items to be saved. Increasing the total transaction lifetime timeout setting of your IBM WebSphere Portal server can address this issue. The total transaction lifetime timeout setting of your subscriber should be at least the same as the syndicator. The total transaction lifetime timeout setting is changed using the IBM WebSphere Application Server administrative console. Go to Application Servers > server > Container Services > Transaction Service.

        I would suggest a value of 600 here

        Also, check your db2diag logs for any errors during this time.
        • mojojojo
          mojojojo
          29 Posts
          ACCEPTED ANSWER

          Re: many java.sql.SQLException (s)

          ‏2012-05-16T14:27:12Z  in response to SystemAdmin
          Thank you Hittupat,

          We do not have any Syndication setup in our Portal.
          And the subscriber is temporarily disabled (was disabled before the upgrade to WPS-6.1.0.6). So i doubt if this is related to Syndication.
          Did you see any specific log entries which made you suggest it is related to Syndication?

          These thread hang & CPU Starvation errors appears during Portal startup, and when i stop the portal, i see these exact threads being recovered.

          ESIConfiguration was also proper. I could not find much details about any problems there.

          Note that: we had also upgraded our DB2 from v9.1-FP7 to v9.1-FP9.
          The JDBC client was not updated. I am checking if there was any update required with jdbc client.

          Thanks for your suggestions.
          • SystemAdmin
            SystemAdmin
            30895 Posts
            ACCEPTED ANSWER

            Re: many java.sql.SQLException (s)

            ‏2012-05-16T14:42:20Z  in response to mojojojo
            Yes, I do see the following exception that suggests that the pending wcm update are made through event logs.

            java.sql.SQLException: DB2 SQL error: SQLCODE: -1035, SQLSTATE: 57019, SQLERRMC: nullDSRA0010E: SQL State = 57019, Error Code = -1,035
            at com.ibm.db2.jcc.c.bb.a(bb.java:2875)
            at com.ibm.db2.jcc.b.bb.q(bb.java:696)
            at com.ibm.db2.jcc.b.bb.p(bb.java:572)
            at com.ibm.db2.jcc.b.bb.l(bb.java:393)
            at com.ibm.db2.jcc.b.bb.d(bb.java:142)
            at com.ibm.db2.jcc.b.b.rc(b.java:1286)
            at com.ibm.db2.jcc.b.b.b(b.java:1210)
            at com.ibm.db2.jcc.b.b.y(b.java:4535)
            at com.ibm.db2.jcc.b.b.b(b.java:745)
            at com.ibm.db2.jcc.b.b.a(b.java:688)
            at com.ibm.db2.jcc.b.b.a(b.java:372)
            at com.ibm.db2.jcc.b.b.<init>(b.java:308)
            at com.ibm.db2.jcc.b.c.<init>(c.java:33)
            at com.ibm.db2.jcc.DB2PooledConnection.<init>(DB2PooledConnection.java:179)
            at com.ibm.db2.jcc.DB2XAConnection.<init>(DB2XAConnection.java:44)
            at com.ibm.db2.jcc.DB2XADataSource.getXAConnection(DB2XADataSource.java:81)
            at com.ibm.db2.jcc.DB2XADataSource.getXAConnection(DB2XADataSource.java:66)
            at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper$1.run(InternalGenericDataStoreHelper.java:956)
            at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118)
            at com.ibm.ws.rsadapter.spi.InternalGenericDataStoreHelper.getPooledConnection(InternalGenericDataStoreHelper.java:993)
            at com.ibm.ws.rsadapter.spi.InternalDB2UniversalDataStoreHelper.getPooledConnection(InternalDB2UniversalDataStoreHelper.java:1703)
            at com.ibm.ws.rsadapter.spi.WSRdbDataSource.getPooledConnection(WSRdbDataSource.java:1704)
            at com.ibm.ws.rsadapter.spi.WSManagedConnectionFactoryImpl.createManagedConnection(WSManagedConnectionFactoryImpl.java:1258)
            at com.ibm.ejs.j2c.FreePool.createManagedConnectionWithMCWrapper(FreePool.java:1992)
            at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1664)
            at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2391)
            at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:940)
            at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:614)
            at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:449)
            at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:418)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.openConnection(EventLogDatabase.java:2505)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.connect(EventLogDatabase.java:314)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.getConnection(EventLogDatabase.java:2518)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.access$500(EventLogDatabase.java:61)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase$16.call(EventLogDatabase.java:3574)
            at com.ibm.workplace.wcm.util.transaction.TransactionManager.doCall(TransactionManager.java:178)
            at com.ibm.workplace.wcm.util.transaction.TransactionManager.call(TransactionManager.java:71)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.entryExists(EventLogDatabase.java:3583)
            at com.ibm.workplace.wcm.services.eventlog.EventLogDatabase.syndicatorUpdatesPending(EventLogDatabase.java:1731)
            at com.ibm.workplace.wcm.services.eventlog.EventLogServiceImpl.syndicatorUpdatesPending(EventLogServiceImpl.java:1215)
            • at com.aptrix.deployment.itemgatherer.ItemChangedTask$ItemChangedWork.updatesPending(ItemChangedTask.java:152)*

            If the syndication is disabled and not required then reset the event logs and restart the server. If this does solve the issue then reupload the systemout.log file.

            http://publib.boulder.ibm.com/infocenter/wpdoc/v6r1/index.jsp?topic=/com.ibm.wp.ent.doc_v6101/wcm/wcm_config_reset_event_log.html
            • mojojojo
              mojojojo
              29 Posts
              ACCEPTED ANSWER

              Re: many java.sql.SQLException (s)

              ‏2012-05-16T17:55:31Z  in response to SystemAdmin
              I did the reset, and then started portal.I did the reset, and then started portal.

              I found the errors about Thread hung again. found the errors about Thread hung again. :|

              > 5/16/12 19:42:00:318 CEST 00000026 ThreadMonitor W WSVR0605W: Thread "WebContainer : 1" (00000046) has been active for 767268 > milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.

              attached the SystemOut.log during Portal start.

              Thank you.
              • SystemAdmin
                SystemAdmin
                30895 Posts
                ACCEPTED ANSWER

                Re: many java.sql.SQLException (s)

                ‏2012-05-16T18:57:34Z  in response to mojojojo
                I reviewed the systemout.log. I don't see hangs related to wcm syndication anymore. However, I do see hang related to "ESIProcessor". Looks like this is coming from the WAS code stream. I see that you are already on WAS version 6.1.0.43 which should already have the ifix PM31557.

                http://www-01.ibm.com/support/docview.wss?uid=swg1PM31557

                I think this problem may need detailed investigation. I suggest that you a open a problem ticket with WAS support.

                Here is the hung thread stack.

                5/16/12 19:41:31:034 CEST 00000011 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl initialize FFDC0009I: FFDC opened incident stream file /portal/WebSphere/profiles/wp_profile/logs/ffdc/WebSphere_Portal_00000011_12.05.16_19.41.31_0.txt
                5/16/12 19:41:31:498 CEST 00000011 ServiceLogger I com.ibm.ws.ffdc.IncidentStreamImpl resetIncidentStream FFDC0010I: FFDC closed incident stream file /portal/WebSphere/profiles/wp_profile/logs/ffdc/WebSphere_Portal_00000011_12.05.16_19.41.31_0.txt
                5/16/12 19:42:00:318 CEST 00000026 ThreadMonitor W WSVR0605W: Thread "WebContainer : 1" (00000046) has been active for 767268 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.
                at java.lang.Object.wait(Native Method)
                at java.lang.Object.wait(Object.java:199)
                at com.ibm.ws.cache.servlet.ESIProcessor.listen(ESIProcessor.java:412)
                at com.ibm.ws.cache.servlet.ESIProcessor.runCommon(ESIProcessor.java:107)
                at com.ibm.ws.cache.servlet.ESIProcessor.run(ESIProcessor.java:83)
                at com.ibm.websphere.servlet.cache.ESIInvalidatorServlet.doPost(ESIInvalidatorServlet.java:48)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:763)
                at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
                at com.ibm.ws.cache.servlet.ServletWrapper.serviceProxied(ServletWrapper.java:282)
                at com.ibm.ws.cache.servlet.CacheHook.handleFragment(CacheHook.java:455)
                at com.ibm.ws.cache.servlet.CacheHook.handleServlet(CacheHook.java:265)
                at com.ibm.ws.cache.servlet.ServletWrapper.service(ServletWrapper.java:262)
                at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1213)
                at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:658)
                at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:526)
                • mojojojo
                  mojojojo
                  29 Posts
                  ACCEPTED ANSWER

                  Re: many java.sql.SQLException (s)

                  ‏2012-05-16T19:18:15Z  in response to SystemAdmin
                  alright.

                  Thanks a lot again, for all the suggestions.
                  • mojojojo
                    mojojojo
                    29 Posts
                    ACCEPTED ANSWER

                    Re: many java.sql.SQLException (s)

                    ‏2012-05-21T16:11:47Z  in response to mojojojo
                    I have raised a PMR with IBM for the ESIProcessor thread hang.

                    And, there are some errors in another portal (attached log). Again, I get them during server startup itself.
                    They seem to be related to some IBM portlets (PA_contentlinker, WCM Tool - Apply Default Content etc)
                    Should i be bothered about these errors?

                    Thank you.
                    • mojojojo
                      mojojojo
                      29 Posts
                      ACCEPTED ANSWER

                      Re: many java.sql.SQLException (s)

                      ‏2012-05-21T16:25:27Z  in response to mojojojo
                      i raised a new thread for this.
                      better to track it separately.

                      https://www.ibm.com/developerworks/forums/thread.jspa?threadID=429124

                      thanks