webMethods Knowlegebase : Document stuck in Broker client queue (1745506)

Problem

A document published by IS1 is subscribed by a trigger on IS2. After publish the document is correctly routed to appropriate trigger client queue on IS2. Strange thing happens here, that document sits in the queue for a very long time and eventually gets processed after several hours. I noticed that sometime after the document is stored in the broker client queue on IS2 a TriggerTask kicks in (in my case it is - TriggerTask: 1:TPWDailyInventoryReportEUPMap.integrations:triggerProcessDailyInventoryReport). This task continues for a very long time; note that the service the trigger is supposed to invoke is not invoked yet because the triggertask has not passed the control to the service yet. Once the trigger task is complete (which takes unusually long time) the associated service executes very quickly.

I do not see any obvious errors in error or server log on IS2. This issue happens on production as well as on development environment. I have taken several thread dumps on DEV server.

Majority of time spent by the trigger task is here,

Integration Server thread dump Java HotSpot(TM) 64-Bit Server VM (20.1-b02-jre1.6.0.12-rc1b1 mixed mode):

Wed Nov 21 00:02:16 CST 2012

"TriggerTask:1:TPWDailyInventoryReportEUPMap.integrations:triggerProcessDailyInventoryReport" Id=490 in RUNNABLE

at java.util.zip.ZipFile.getEntry(Native Method)

at java.util.zip.ZipFile.getEntry(ZipFile.java:162)

at java.util.jar.JarFile.getEntry(JarFile.java:208)

at java.util.jar.JarFile.getJarEntry(JarFile.java:191)

at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:757)

at sun.misc.URLClassPath.getResource(URLClassPath.java:169)

at java.net.URLClassLoader$1.run(URLClassLoader.java:194)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

at java.lang.ClassLoader.loadClass(ClassLoader.java:295)

at com.wm.app.b2b.server.ServerClassLoader.loadClass(ServerClassLoader.java:797)

at com.wm.app.b2b.server.ServerClassLoader.loadClass(ServerClassLoader.java:736)

at com.wm.app.b2b.server.ServerClassLoader.findClassInDepChain(ServerClassLoader.java:715)

at com.wm.app.b2b.server.ServerClassLoader.findClass(ServerClassLoader.java:466)

at java.lang.ClassLoader.loadClass(ClassLoader.java:306)

at com.wm.app.b2b.server.ServerClassLoader.loadClass(ServerClassLoader.java:787)

at com.wm.app.b2b.server.ServerClassLoader.loadClass(ServerClassLoader.java:736)

at com.wm.util.coder.IDataBinCoder.loadClass(IDataBinCoder.java:1373)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:775)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:990)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:990)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder._decode(IDataBinCoder.java:818)

at com.wm.util.coder.IDataBinCoder.decode(IDataBinCoder.java:307)

at com.wm.broker.coder.BrokerCoder.decodeFromBytes(BrokerCoder.java:320)

at com.wm.app.b2b.server.dispatcher.BrokerEventCoder.decodeToIData(BrokerEventCoder.java:255)

at com.wm.app.b2b.server.dispatcher.BrokerEventCoder$BrokerEventDispatcherMessageCoder.decodeToIData(BrokerEventCoder.java:348)

at com.wm.app.b2b.server.dispatcher.trigger.TriggerTask.run(TriggerTask.java:207)

at com.wm.util.pool.PooledThread.run(PooledThread.java:131)

at java.lang.Thread.run(Thread.java:662)

Started : Tue, 20 Nov 2012 22:17:02

While the trigger task continues to run I see 1 document processing thread associated with the trigger “TPWDailyInventoryReportEUPMap.integrations:triggerProcessDailyInventoryReport” in my case. That goes to 0 when the processing completes.

Size of the document in the trigger client queue is 1732096 bytes

Version of development IS 8.2.2

Version of production IS 6.5.3

Version of broker on development and production is 8.2.2.0.259 100511

Question:

1. Why does it take so long to retrieve the document from Broker and invoke associated service on IS?