At my company we are exploring purequery as an option to allow access to our DB2 databases running on zSeries directly from java code running in application servers.
I've been doing some testing, currently using standard java applications (ie, not running under an Application Server), testing accesses both against the zSeries DB2 (version 9.1) and against instances of DB2 Express-C running under Linux (Ubuntu Linux 11.04). I've observed that sometimes the applications just hang and I've got to kill them. I've observed that only against the LUW DB2 (which does not preclude it happening also against the z machine, since the hangs are intermitent). I've got traces at the driver and the network level and it seems like the application just stop talking to the DB2 server after successfully stablishing connection. It definitely DOES NOT HAPPEN unless the pureQuery runtime is active, and it tends to happen a little bit more (totally unscientifical, just my gut feeling here) when the executionMode is set to STATIC.
Anyone else is having the same problem?
BTW; I'm quite impressed with the product, and I'm heavily recommending my management to use it.
This topic has been locked.
10 replies Latest Post - 2012-01-11T18:14:32Z by jaijeet
Pinned topic Intermitent hangups using purequery 3.1 under Linux
Answered question This question has been answered.
Unanswered question This question has not been answered yet.
Updated on 2012-01-11T18:14:32Z at 2012-01-11T18:14:32Z by jaijeet
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-06T22:10:22Z in response to jguillaumesHi
Are you using the Client Optimization flavor of pureQuery or using the Data API (with annotated methods) ?
Do you have the JCC Driver traces when this hang occurred ?
you mentioned that hang does not occur when pureQuery runtime is not active - does it mean when pdq.jar is not in CLASSPATH the hang does not occur ?
Please provide the JCC Driver / pureQuery versions you are using.
SystemAdmin 110000D4XK90 PostsACCEPTED ANSWER
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-06T23:37:54Z in response to jaijeetHi Jordi,
In addition to the questions that Jaijeet asked, we're also interested in what it looks like from the DB2 side when the hang occurs. Can you get a DB2 snapshot see what the state of the connection is? We're curious to see what the last statements are that it executed, and whether it left the connection in an uncommitted state. Try this command:
db2 get snapshot for all applications
or if you can get the appl-id or agent-id for the hanging app (see LIST APPLICATIONS command), you can ask for just information on that connection:
db2 get snapshot for application applid appl-id
Also, a few more questions:
You said it was intermittent, but when it does hang, is it always in the same place in the application?
I was not certain from your description whether the Java application was running standalone, or if it was running in an Application Server container such as Websphere or Tomcat. If so, we would be interested in seeing the app server log to see if there is anything unusual reported there.
When this hangs, is there a lot of other activity going on in the app server or on the database?
I'm happy to hear that you're mostly impressed with pureQuery so far. Thanks for any additional info you can provide.
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-07T11:07:29Z in response to SystemAdminI'm attaching the output of the get snapshot command. The database is named TLE and the application is TLEJM60. There is no other workload running, this is my own development machine.
It hangs just at the FIRST attempt to talk to DB2, after the connection has been stablished. I can't debug it (because it DOES NOT HAPPEN when I run the application under Eclipse debugger) but using the old method of making the app verbose I've been able to pinpoint the hang at a setTransactionIsolation() call. If I remove this, it stops at the following setAutoCommit(), and if I remove this too it stops at the first prepareStatement().
It is a standalone java application. I'm just beginning to explore the product. :)
There is no other activity in the database when I do this test. It's a DB2 EXP-C instance I've got in my development machine.
I'm attaching the pdq trace in another reply.
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-10T01:30:09Z in response to jguillaumesHi
Thank you sending the traces. It is not clear why the hang occurred but could you please try with this property set
This property disables service threads in PQ runtime. There could be some timing issue in the service threads and
as you observed goes away when debugger is turned on.
Service threads are created to fetch the xml from the Repository as we cannot use the users connection/transaction
to fetch the capture xml from the Repository.
In yr case the capture file resides locally on the disk and Repository is not in use so we do not need
the service threads.
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-10T17:51:39Z in response to jaijeetHi,
It seems that setting that property fixes the issue. Of course, in a production environment we should be using a repository and not an XML local file, so I'm not sure if the fix would be sufficient.
As for the actual problem, I've been able to get the stack dump of the application in its frozen state using the remote debugging facility of the JVM. There are two active tracks (their appear suspended in the trace because I suspended them), which are these ones:
Thread main (Suspended)
Object.wait(long) line: not available native method
Thread(Object).wait() line: 485
PdqServiceThreadProvider.executeOnServiceThreadIfNeeded(PdqServiceThreadProvider$ServiceThreadRequest, ConnectionExecutionHandler, boolean, Object...) line: 286
DB2ConnectionExecutionHandler(ConnectionExecutionHandler).invoke(String, Method, Object) line: 1483
ProxiedDB2ConnectionInvocationHandler(ProxiedJdbcConnectionInvocationHandler).invokeMethod(Method, Object, String) line: 258
ProxiedDB2ConnectionInvocationHandler(ProxiedJdbcConnectionInvocationHandler).invoke(Object, Method, Object) line: 197
$Proxy0.setTransactionIsolation(int) line: not available
DriverTLEJM60S.main(String) line: 37
Object.wait(long) line: not available native method
TaskQueue(Object).wait() line: 485
TimerThread.mainLoop() line: 483
TimerThread.run() line: 462
One more detail which (perhaps) could be significant. I'm running these tests under a virtual machine. Actually, under two different virtual machines: VMWare Fusion and Parallels Desktop under Mac OS. The hangups appear on both. I don't have any "bare metal" linux machine available, so I can't tell you if this is a virtulization issue (it lools like a remote possibility).
Thanks for your great support!
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-12T15:03:21Z in response to jguillaumesHi
Thank you sending the traces. looking at the traces we feel the problem occurs due a recently fixed defect.
The short description of the problem is that we start the Service Provider Thread to read the XML file (file system or Repository), once we have started the thread we wait for the Service Provider Thread to finish, once it finishes the application thread continues execution. It turns out there is a very small window where the Service Provider Thread can finish before the application thread waits for it, if this happens the application thread waits forever. The environment of Linux and Sun JRE makes this window much bigger and we need to protect against this.
I am attaching a development build which has the fix. Please test with this build.
Please note that that build is a dev build only and is not be distributed or used in production
Before using verify the version using the DataVesion utility
with pdq.jar in CLASSPATH run
IBM InfoSphere Optim pureQuery Runtime 22.214.171.124 build 3.110.28
Edited by: admin on Jan 12, 2012 10:03 AMUpdated on 2012-01-12T15:03:21Z at 2012-01-12T15:03:21Z by jaijeet
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-11T18:14:32Z in response to jguillaumesHi
That was good to hear that the new pdq.jar fixed the problem.
Please do keep us informed about yr progress on testing testing pureQuery
and any feedback will be appreciated.
Re: Intermitent hangups using purequery 3.1 under Linux2012-01-07T10:59:33Z in response to jaijeetThanks for your fast reply :)
I'm attaching the jcc trace to this reply. I've set the tracelevel to -1.
For "PDQ inactive" I should had to say "executionMode=DYNAMIC". There are no hangups unless I select STATIC in the properties file. If you want another ghost to chase, it does not happen if I run the application under control of the Eclipse debugger (yeah, that's weird, I know).
The versions I'm using are:
- IBM Data Server driver for JDBC and SQLJ at release 4.8.87
- IBM JDK implementation version 2.4 (the one distributed with Data Studio 3.1)
- IBM InphoSphere Optim pureQuery runtime 126.96.36.199 build 3.10.56
(All of these are taken from the pdq trace, which I'll attach in another reply).
As for WHEN does it happen. Well, it hangs at the FIRST time the application tries to talk to db2. In this case, it happens in a setIsolationLevel() call. If I comment out that call, it hangs at a setAutoCommit() call. If I supress that one too, it hangs at the first prepareStatement().