PI17533: JMS MESSAGE PRODUCER AND CONSUMER APPLICATIONS FAIL TO PRODUCE AND CONSUME MESSAGES FROM SERVICE INTEGRATION BUS DESTINATION

Fixes are available

APAR status

Closed as program error.

Error description

JMS application is configured to send messages to Websphere
Application Server v7.0.x Service Integration Bus destination
and an MDB is configured to consume messages from the
destination.The message producer and consumer application
suddenly stop producing and consuming messages from the
destination.Javacores collected from the server hosting the
applications and the server hosting the messaging engine
show the threads in blocked state with the following stack:

On the server hosting the messaging engine,7 out of 8
SpillDispatcher threads were hung while adding messages to
the filestore:

at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at
com/ibm/ws/objectManager/AbstractSingleFileObjectStore.reserve(A
bstractSingleFileObjectStore.java:718(Compiled Code))
at
com/ibm/ws/objectManager/ManagedObject.reserveSpaceInStore(Manag
edObject.java:663(Compiled Code))
at
com/ibm/ws/objectManager/ManagedObject.getSerializedBytes(Manage
dObject.java:637(Compiled Code))
at
com/ibm/ws/objectManager/InternalTransaction.add(InternalTransac
tion.java:780(Compiled Code))
at
com/ibm/ws/objectManager/Transaction.add(Transaction.java:243(Co
mpiled Code))
at
com/ibm/ws/sib/msgstore/persistence/objectManager/PersistableImp
l.addToStore(PersistableImpl.java:454(Compiled Code))
at
com/ibm/ws/sib/msgstore/persistence/objectManager/BatchingContex
tImpl.insert(BatchingContextImpl.java:205(Compiled Code))
at
com/ibm/ws/sib/msgstore/task/AddTask.persist(AddTask.java:371(Co
mpiled Code))
at
com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S
pillDispatcherThread.writeBatch(SpillDispatcher.java:1669(Compil
ed Code))
at
com/ibm/ws/sib/msgstore/persistence/dispatcher/SpillDispatcher$S
pillDispatcherThread.run(SpillDispatcher.java:711(Compiled
Code))
at java/lang/Thread.run(Thread.java:738(Compiled Code))


On the server hosting the message producer and consumer
applications 90+ threads were found with the following stack:

Consumer threads:

at java/lang/Object.wait(Native Method)
at java/lang/Object.wait(Object.java:167(Compiled Code))
at
com/ibm/ws/sib/jfapchannel/impl/ExchangeReceiveListener.waitToCo
mplete(ExchangeReceiveListener.java:227(Compiled Code))
at
com/ibm/ws/sib/jfapchannel/impl/ConversationImpl.exchange(Conver
sationImpl.java:988(Compiled Code))
at
com/ibm/ws/sib/comms/common/JFAPCommunicator.jfapExchange(JFAPCo
mmunicator.java:386(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy.performReceive(
ConsumerSessionProxy.java:914(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy._receiveWithWai
t(ConsumerSessionProxy.java:825(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConsumerSessionProxy.receiveWithWait
(ConsumerSessionProxy.java:733(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receiveInboundMes
sage(JmsMsgConsumerImpl.java:1114(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgConsumerImpl.receive(JmsMsgCon
sumerImpl.java:481(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueReceiverImpl.receive(JmsQueu
eReceiverImpl.java(Compiled Code))

Producer threads:

at
com/ibm/ws/sib/comms/client/ConnectionProxy._createProducerSessi
on(ConnectionProxy.java:1007(Compiled Code))
at
com/ibm/ws/sib/comms/client/ConnectionProxy.createProducerSessio
n(ConnectionProxy.java:932(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsMsgProducerImpl.<init>(JmsMsgProd
ucerImpl.java:369(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueSenderImpl.<init>(JmsQueueSe
nderImpl.java:60(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsQueueSessionImpl.instantiateProdu
cer(JmsQueueSessionImpl.java:224(Compiled Code))
at
com/ibm/ws/sib/api/jms/impl/JmsSessionImpl.createProducer(JmsSes
sionImpl.java:866(Compiled Code))

Local fix

Problem summary

****************************************************************
* USERS AFFECTED:  Users of the default messaging provider     *
*                  for IBM WebSphere Application Server V7.0,  *
*                  V8.0 and V8.5                               *
****************************************************************
* PROBLEM DESCRIPTION: Message producer application suddenly   *
*                      stops producing messages to the         *
*                      destination because dispacher threads   *
*                      are hung.                               *
****************************************************************
* RECOMMENDATION:                                              *
****************************************************************
When persisting messages into the filestore, an object manager
needs to ensure free space in the filestore to store the
messages. So the "SpillDispatcher" thread is indefinitely
waiting in AbstractSingleFileObjectStore.reserve(). Another
thread notifies the waiting thread based on a condition which
is controlled by a boolan variable which is not thread safe.
So the waiting "SpillDispatcher" thread may not get notified
and waits forever and this may eventually happen to all the
dispacther threads.
When all the dispacther threads are hung, the message producer
hangs as there are no dispacther threads to persist messages.
Consumer application waits on receiveWithWait() because there
are no messages in the destination.
This is a rare problem due to timing especially when there are
more CPUs. This problem is applicable if file store is being
used as a message store.

Problem conclusion

Source code has been modified to ensure that the boolean
variable is thread safe.

The fix for this APAR is currently targeted for inclusion in
fix packs 7.0.0.37, 8.0.0.10, and 8.5.5.4. Please refer to the
Recommended Updates page for delivery information:
http://www.ibm.com/support/docview.wss?rs=180&uid=swg27004980

Temporary fix

Comments

APAR Information

APAR number
PI17533
Reported component name
WAS SIB & SIBWS
Reported component ID
620800101
Reported release
300
Status
CLOSED PER
PE
NoPE
HIPER
NoHIPER
Special Attention
NoSpecatt
Submitted date
2014-05-08
Closed date
2014-08-18
Last modified date
2014-08-18

APAR is sysrouted FROM one or more of the following:
APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name
WAS SIB & SIBWS
Fixed component ID
620800101

Applicable component levels

R800 PSY
UP
R850 PSY
UP

[{"Business Unit":{"code":"BU059","label":"IBM Software w\/o TPS"},"Product":{"code":"SSEQTP","label":"WebSphere Application Server"},"Platform":[{"code":"PF025","label":"Platform Independent"}],"Version":"7.0","Line of Business":{"code":"LOB45","label":"Automation"}}]

Document Information

Modified date:
28 April 2022

Tips

PI17533: JMS MESSAGE PRODUCER AND CONSUMER APPLICATIONS FAIL TO PRODUCE AND CONSUME MESSAGES FROM SERVICE INTEGRATION BUS DESTINATION

Fixes are available

Subscribe

APAR status

Closed as program error.

Error description

Local fix

Problem summary

Problem conclusion

Temporary fix

Comments

APAR Information

APAR number

Reported component name

Reported component ID

Reported release

Status

PE

HIPER

Special Attention

Submitted date

Closed date

Last modified date

APAR is sysrouted FROM one or more of the following:

APAR is sysrouted TO one or more of the following:

Fix information

Fixed component name

Fixed component ID

Applicable component levels

R800 PSY

R850 PSY

Document Information

Share your feedback

Need support?