20
Have you have seen message WTRN0006W time after time in the SystemOut.log?
WTRN0006W: Transaction {0} has timed out after {1} seconds.
And if you follow the information center you would have seen:
Explanation The transaction service has timed out (rolled back) the transaction identified in the message.
Action To reduce future timeout occurrences, increase the transaction timeout values.
Well, first of all, WTRN0006W in itself is not always the problem. It is just the symptom and to tackle the WTRN0006W you should identify the problem before you decide to alter the transaction timeout values. Let's look at some common problems that lead to transaction timeouts. In general, try to isolate the application flow which ends in a transaction timeout.
1. How is the system performance? Do you see CPU is highly utilized than normal? Do you observe memory paging? Are you running out of disk space? The problem could be resource bottleneck. Ensure you have done proper hardware capacity planning before you go into production. It would be worthwhile to investigate symptom-wise and resolve the bottleneck before adjusting the transaction settings.
2. Do you see WSVR0605W and WTRN0041I in conjunction with WTRN0006W? Look out for the WSVR0605W messages first. The message notifies you that a thread has been active for a (very) long time and could be hung. This hang could be the cause of transaction timing out and then eventually transaction rolling back due to the timeout. You will observe the same thread reporting the WTRN0006W and WTRN0041I.
Look at the example below, from WebSphere Application Server V7.0, the thread 00000006 is reporting a transaction has timed out after the default timeout period. The thread that was associated with the transaction is listed out, as well as, its stack trace. Investigate the stack trace to identify the source of the hang. In our example, a JDBC driver socket connection is seen to have hung in an ORB thread. Note that the message is WTRN0124I.
[2/4/11 16:38:56:674 GMT+05:30] 00000006 TimeoutManage I WTRN0006W:
Transaction
0000012DF05A77220000000100000583B2AEFB85A9BC1C37D5059296FA62C5507319FFE9
0000012DF05A77220000000100000583B2AEFB85A9BC1C37D5059296FA62C5507319FFE9
00000001 has timed out after 120 seconds.[2/4/11 16:38:56:677 GMT+05:30] 00000006 TimeoutManage I WTRN0124I:
When the timeout occurred the thread with which the transaction is, or
was most recently, associated was Thread[ORB.thread.pool : 21,5,main].
The stack trace of this thread when the timeout occurred was:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:141)
com.ibm.db2.jcc.t4.z.b(z.java:200)
com.ibm.db2.jcc.t4.z.c(z.java:260)
com.ibm.db2.jcc.t4.z.c(z.java:371)3. Are you running into a JVM error condition? Look out for OutOfMemory messages in the SystemOut logs and check whether heapdumps are being generated in the <profile> directory. An OutOfMemory condition will halt the processing of threads until it is handled. Transactions in-flight are bound to timeout. In a network deployment setup ensure that you monitor all the clusters for JVM health. For instance, you might get transaction timeouts while SCA tries to connect to a messaging engine and you may not find any further messages in the SCA cluster. Meanwhile the messaging cluster will be undergoing the bottleneck.
4. Is your target service available? Check whether the target service is still available when making asynchronous SCA invokes. If the target service is unavailable the transaction in which the invoke is called will timeout and an error will be returned to the caller. Revisit the SCA invocation styles if your target service does not guarantee a constant uptime. You may not want to change the total transaction timeout for this particular invoke because the timeout changes will apply to all the applications running on the server. Asynchronous invocations provides request and response expiration qualifiers that can be configured at invocation level.
If you are making calls to a database server or an external messaging provider ensure that the thread pool settings are adequate so applications do not timeout waiting for free connections. You should check whether it is these calls that are timed out due to bottleneck at provider end.
5. Is it important to note that, WTRN0006W is reported after the transaction has timed out. Enabling the transaction traces at this juncture might not help debug the problem. It is therefore crucial that you preserve the historical logs (and not just the recent SystemOut.log). In case you want to enable transaction traces ensure that the problem is reproducible. Keep the traces enabled until the problem is captured. Ensure that you have appropriately set the number of historical files and trace log size, so that trace logs are not rolled over (overwritten).
And now the recommended reading list:
TroubleShooting: Java Transaction Service (JTS) problems (here)
Listen to the WSTE webcast replay on Transactions in WebSphere Application Server: Overview and Problem Determination by David Tiler, Level 2 Support Analyst (here)
Tuning transaction service settings (here)
Finally, are you a fan of IBM Support Assistant (ISA)? If yes, you will be happy to know that IBM Guided Activity Assistant (IGAA) which is a part of (ISA) has a new activity for troubleshooting transaction timeouts (here)
