Using streaming queues to store a history of messages

You can use streaming queues to retain a history of messages for a limited period of time, and you achieve this by configuring the CAPEXPRY attribute on the queue that messages are streamed to.

Introduction

When messages are streamed from one queue to another, any expiry value set on the message is reset to a value of MQEI_UNLIMITED for the duplicate copy. By default, this results in a steady build up of messages on the queue you are streaming to, if there is no application consuming them.

In this scenario you want to keep a copy of the messages for a limited period of time so that you are able to access them. For example, if the original message has been accidentally removed by a consuming application.

It is not feasible to keep a copy of every message indefinitely, and to prevent the queue you are streaming to from becoming full there are two options:

Run an application to remove the messages every so often
Configure the messages with an expiry, causing them to be removed by IBM® MQ after a period of time.

The second option can be much more convenient as it does not require you to run and maintain an application, just to prevent the queue from filling up.

Configuring CAPEXPRY

The Enforcing lower expiration times topic describes how to configure CAPEXPRY on a queue. For this scenario, you must set the CAPEXPRY attribute on the queue to which you are streaming messages.

Note: You do not need to change the value of the CAPEXPRY attribute on the queue that messages are being streamed from.

Choose a suitable expiry time for the duplicate messages, taking into account the following considerations:

How long you might need access to the messages for
What the MAXDEPTH attribute of the queue needs to be, based on the rate of messages being put to the original queue
How much storage you need in order to store the duplicate messages.
This might require consideration of the queue manager file system size, and the MAXFSIZE attribute of the queue.

Accessing the duplicate messages if they are needed

If a problem should occur that requires you to access and recover some or all the duplicate messages, use the dmpmqmsg command, to help access those messages.

dmpmqmsg has options for:

Reading the messages from the queue and writing a copy to file to access later
Reading messages from a file and writing them back onto a queue for applications to consume

It is possible to edit the message headers once dmpmqmsg has written them to a file. For example, you could reset the expiry of the messages back to MQEI_UNLIMITED, before dmqmqmsg puts them back onto a queue for processing, by changing the EXP value of every message in the file to -1.

Performance considerations

There is a small cost to streaming duplicate messages to another queue and expiring them when they are no longer required. However, the cost is much smaller than manually putting a copy to a second queue and having an application destructively remove them after a period of time. The Streaming Queues Performance Report gives more information about the performance of this scenario.