Topic
IC4NOTICE: developerWorks Community will be offline May 29-30, 2015 while we upgrade to the latest version of IBM Connections. For more information, read our upgrade FAQ.
10 replies Latest Post - ‏2012-01-11T18:14:32Z by jaijeet
jguillaumes
jguillaumes
10 Posts
ACCEPTED ANSWER

Pinned topic Intermitent hangups using purequery 3.1 under Linux

‏2012-01-06T21:26:51Z |
Hello,

At my company we are exploring purequery as an option to allow access to our DB2 databases running on zSeries directly from java code running in application servers.

I've been doing some testing, currently using standard java applications (ie, not running under an Application Server), testing accesses both against the zSeries DB2 (version 9.1) and against instances of DB2 Express-C running under Linux (Ubuntu Linux 11.04). I've observed that sometimes the applications just hang and I've got to kill them. I've observed that only against the LUW DB2 (which does not preclude it happening also against the z machine, since the hangs are intermitent). I've got traces at the driver and the network level and it seems like the application just stop talking to the DB2 server after successfully stablishing connection. It definitely DOES NOT HAPPEN unless the pureQuery runtime is active, and it tends to happen a little bit more (totally unscientifical, just my gut feeling here) when the executionMode is set to STATIC.

Anyone else is having the same problem?

BTW; I'm quite impressed with the product, and I'm heavily recommending my management to use it.
Updated on 2012-01-11T18:14:32Z at 2012-01-11T18:14:32Z by jaijeet
  • jaijeet
    jaijeet
    5 Posts
    ACCEPTED ANSWER

    Re: Intermitent hangups using purequery 3.1 under Linux

    ‏2012-01-06T22:10:22Z  in response to jguillaumes
    Hi

    Are you using the Client Optimization flavor of pureQuery or using the Data API (with annotated methods) ?

    Do you have the JCC Driver traces when this hang occurred ?
    you mentioned that hang does not occur when pureQuery runtime is not active - does it mean when pdq.jar is not in CLASSPATH the hang does not occur ?
    Please provide the JCC Driver / pureQuery versions you are using.

    thanks.
    • SystemAdmin
      SystemAdmin
      90 Posts
      ACCEPTED ANSWER

      Re: Intermitent hangups using purequery 3.1 under Linux

      ‏2012-01-06T23:37:54Z  in response to jaijeet
      Hi Jordi,
      In addition to the questions that Jaijeet asked, we're also interested in what it looks like from the DB2 side when the hang occurs. Can you get a DB2 snapshot see what the state of the connection is? We're curious to see what the last statements are that it executed, and whether it left the connection in an uncommitted state. Try this command:

      db2 get snapshot for all applications

      or if you can get the appl-id or agent-id for the hanging app (see LIST APPLICATIONS command), you can ask for just information on that connection:

      db2 get snapshot for application applid appl-id

      Also, a few more questions:

      You said it was intermittent, but when it does hang, is it always in the same place in the application?

      I was not certain from your description whether the Java application was running standalone, or if it was running in an Application Server container such as Websphere or Tomcat. If so, we would be interested in seeing the app server log to see if there is anything unusual reported there.

      When this hangs, is there a lot of other activity going on in the app server or on the database?

      I'm happy to hear that you're mostly impressed with pureQuery so far. Thanks for any additional info you can provide.

      Regards,
      Bill Bireley
      • jguillaumes
        jguillaumes
        10 Posts
        ACCEPTED ANSWER

        Re: Intermitent hangups using purequery 3.1 under Linux

        ‏2012-01-07T11:07:29Z  in response to SystemAdmin
        I'm attaching the output of the get snapshot command. The database is named TLE and the application is TLEJM60. There is no other workload running, this is my own development machine.

        It hangs just at the FIRST attempt to talk to DB2, after the connection has been stablished. I can't debug it (because it DOES NOT HAPPEN when I run the application under Eclipse debugger) but using the old method of making the app verbose I've been able to pinpoint the hang at a setTransactionIsolation() call. If I remove this, it stops at the following setAutoCommit(), and if I remove this too it stops at the first prepareStatement().

        It is a standalone java application. I'm just beginning to explore the product. :)

        There is no other activity in the database when I do this test. It's a DB2 EXP-C instance I've got in my development machine.

        I'm attaching the pdq trace in another reply.
        • jguillaumes
          jguillaumes
          10 Posts
          ACCEPTED ANSWER

          Re: Intermitent hangups using purequery 3.1 under Linux

          ‏2012-01-07T11:08:09Z  in response to jguillaumes
          PDQ trace, as said before.
          • jaijeet
            jaijeet
            5 Posts
            ACCEPTED ANSWER

            Re: Intermitent hangups using purequery 3.1 under Linux

            ‏2012-01-10T01:30:09Z  in response to jguillaumes
            Hi

            Thank you sending the traces. It is not clear why the hang occurred but could you please try with this property set

            pdq.cmx.controllerDiscoveryInterval=-2

            This property disables service threads in PQ runtime. There could be some timing issue in the service threads and
            as you observed goes away when debugger is turned on.

            Service threads are created to fetch the xml from the Repository as we cannot use the users connection/transaction
            to fetch the capture xml from the Repository.
            In yr case the capture file resides locally on the disk and Repository is not in use so we do not need
            the service threads.

            Thanks,
            jaijeet
            • jguillaumes
              jguillaumes
              10 Posts
              ACCEPTED ANSWER

              Re: Intermitent hangups using purequery 3.1 under Linux

              ‏2012-01-10T17:51:39Z  in response to jaijeet
              Hi,

              It seems that setting that property fixes the issue. Of course, in a production environment we should be using a repository and not an XML local file, so I'm not sure if the fix would be sufficient.

              As for the actual problem, I've been able to get the stack dump of the application in its frozen state using the remote debugging facility of the JVM. There are two active tracks (their appear suspended in the trace because I suspended them), which are these ones:

              Thread main (Suspended)
              Object.wait(long) line: not available native method
              Thread(Object).wait() line: 485
              PdqServiceThreadProvider.executeOnServiceThreadIfNeeded(PdqServiceThreadProvider$ServiceThreadRequest, ConnectionExecutionHandler, boolean, Object...) line: 286
              DB2ConnectionExecutionHandler(ConnectionExecutionHandler).invoke(String, Method, Object[]) line: 1483
              ProxiedDB2ConnectionInvocationHandler(ProxiedJdbcConnectionInvocationHandler).invokeMethod(Method, Object[], String) line: 258
              ProxiedDB2ConnectionInvocationHandler(ProxiedJdbcConnectionInvocationHandler).invoke(Object, Method, Object[]) line: 197
              $Proxy0.setTransactionIsolation(int) line: not available
              DriverTLEJM60S.main(String[]) line: 37
              Daemon Thread ControlDataSourcePool@1722456 (Suspended)
              Object.wait(long) line: not available native method
              TaskQueue(Object).wait() line: 485
              TimerThread.mainLoop() line: 483
              TimerThread.run() line: 462

              One more detail which (perhaps) could be significant. I'm running these tests under a virtual machine. Actually, under two different virtual machines: VMWare Fusion and Parallels Desktop under Mac OS. The hangups appear on both. I don't have any "bare metal" linux machine available, so I can't tell you if this is a virtulization issue (it lools like a remote possibility).

              Thanks for your great support!
              • jaijeet
                jaijeet
                5 Posts
                ACCEPTED ANSWER

                Re: Intermitent hangups using purequery 3.1 under Linux

                ‏2012-01-12T15:03:21Z  in response to jguillaumes
                Hi

                Thank you sending the traces. looking at the traces we feel the problem occurs due a recently fixed defect.

                To Summarize:
                The short description of the problem is that we start the Service Provider Thread to read the XML file (file system or Repository), once we have started the thread we wait for the Service Provider Thread to finish, once it finishes the application thread continues execution. It turns out there is a very small window where the Service Provider Thread can finish before the application thread waits for it, if this happens the application thread waits forever. The environment of Linux and Sun JRE makes this window much bigger and we need to protect against this.

                I am attaching a development build which has the fix. Please test with this build.

                Please note that that build is a dev build only and is not be distributed or used in production

                Before using verify the version using the DataVesion utility
                with pdq.jar in CLASSPATH run
                java com.ibm.pdq.tools.DataVersion
                IBM InfoSphere Optim pureQuery Runtime 3.1.1.0 build 3.110.28

                thanks.
                jaijeet

                Edited by: admin on Jan 12, 2012 10:03 AM
                Updated on 2012-01-12T15:03:21Z at 2012-01-12T15:03:21Z by jaijeet
                • jguillaumes
                  jguillaumes
                  10 Posts
                  ACCEPTED ANSWER

                  Re: Intermitent hangups using purequery 3.1 under Linux

                  ‏2012-01-11T10:22:35Z  in response to jaijeet
                  This new jar seems to fix completely the problem. Thank you very much.
                  • jaijeet
                    jaijeet
                    5 Posts
                    ACCEPTED ANSWER

                    Re: Intermitent hangups using purequery 3.1 under Linux

                    ‏2012-01-11T18:14:32Z  in response to jguillaumes
                    Hi

                    That was good to hear that the new pdq.jar fixed the problem.

                    Please do keep us informed about yr progress on testing testing pureQuery
                    and any feedback will be appreciated.

                    Thanks,
                    jaijeet
    • jguillaumes
      jguillaumes
      10 Posts
      ACCEPTED ANSWER

      Re: Intermitent hangups using purequery 3.1 under Linux

      ‏2012-01-07T10:59:33Z  in response to jaijeet
      Thanks for your fast reply :)

      I'm attaching the jcc trace to this reply. I've set the tracelevel to -1.

      For "PDQ inactive" I should had to say "executionMode=DYNAMIC". There are no hangups unless I select STATIC in the properties file. If you want another ghost to chase, it does not happen if I run the application under control of the Eclipse debugger (yeah, that's weird, I know).

      The versions I'm using are:

      • IBM Data Server driver for JDBC and SQLJ at release 4.8.87
      • IBM JDK implementation version 2.4 (the one distributed with Data Studio 3.1)
      • IBM InphoSphere Optim pureQuery runtime 3.1.0.0 build 3.10.56

      (All of these are taken from the pdq trace, which I'll attach in another reply).

      As for WHEN does it happen. Well, it hangs at the FIRST time the application tries to talk to db2. In this case, it happens in a setIsolationLevel() call. If I comment out that call, it hangs at a setAutoCommit() call. If I supress that one too, it hangs at the first prepareStatement().