Managing log files

Allocate sufficient space for your log files. For linear logging, you can delete old log files when they are no longer required.

Information specific to circular logging

If you are using circular logging, ensure that there is sufficient space to hold the log files when you configure your system (see LogDefaults stanza of the mqs.ini file and Log stanza of the qm.ini file). The amount of disk space used by the log does not increase beyond the configured size, including space for secondary files to be created when required.

Information specific to linear logging

If you are using a linear log, the log files are added continually as data is logged, and the amount of disk space used increases with time. If the rate of data being logged is high, disk space is used rapidly by new log files.

Over time, the older log files for a linear log are no longer required to restart the queue manager or to perform media recovery of any damaged objects. The following methods determine which log files are still required:

Logger event messages

When a significant event occurs, for example a record media image, logger event messages are generated. The contents of logger event messages specify the log files that are still required for queue manager restart, and media recovery. For more information about logger event messages, see Logger events

Queue manager status

Running the MQSC command, DISPLAY QMSTATUS, or the PCF command, Inquire Queue Manager Status, returns queue manager information, including details of the required log files. For more information about MQSC commands, see Script (MQSC) Commands, and for information about PCF commands, see Automating administration tasks.

Queue manager messages

Periodically, the queue manager issues a pair of messages to indicate which of the log files are needed:

Message AMQ7467I gives the name of the oldest log file required to restart the queue manager. This log file and all newer log files must be available during queue manager restart.
Message AMQ7468I gives the name of the oldest log file needed for media recovery.

To determine "older" and "newer" log files, use the log file number rather than the modification times applied by the file system.

Information applicable to both types of logging

Only log files required for queue manager restart, active log files, are required to be online. Inactive log files can be copied to an archive medium such as tape for disaster recovery, and removed from the log directory. Inactive log files that are not required for media recovery can be considered as superfluous log files. You can delete superfluous log files if they are no longer of interest to your operation.

If any log file that is needed cannot be found, operator message AMQ6767E is issued. Make the log file, and all subsequent log files, available to the queue manager and try the operation again.

Cleaning log extents automatically - linear logging only

From IBM® MQ 9.0.2 you have the option to use automatic management of linear log extents no longer required for recovery.

You use the LogManagement attribute in the Log stanza of the qm.ini file, or by using the IBM MQ Explorer, to set up automatic management. See Log stanza of the qm.ini file for more information.

See the LOG parameter of DISPLAY QMSTATUS for more details about the operation of the log, and the following commands for using the log:

Taking media images automatically - linear logging only

From IBM MQ 9.0.2 there is an overall switch to control whether the queue manager automatically writes media images, the default being that the switch has not been set.

You can control whether automatic media imaging occurs, and the frequency of the process, by using the following queue manager attributes:

IMGSCHED: Whether the queue manager writes media images automatically
IMGINTVL: Frequency for writing media images, in minutes
IMGLOGLN: Megabytes of log written since previous media image of an object.

If you have a critical time during the day when workload is very heavy and you want to be sure that system throughput is not impacted by taking automatic media images, you might wish to temporarily switch off automatic media imaging by setting IMGSCHED(MANUAL).

You can switch IMGSCHED at any time during the workload.

Attention: MEDIALOG will not be moved forward if you are not taking media images, so you need to, either be archiving extents, or be sure you have sufficient disk space.

You can also control automatic and manual media images for other user-defined objects:

Authentication information
Channel
Client connection
Listener
Namelist
Process
Alias queue
Local queue
Service
Topic

For internal system objects, such as the object catalog and the queue manager object, the queue manager automatically writes media images as appropriate.

See ALTER QMGR for more information on the attributes.

You can also enable or disable automatic and manual media images for local and permanent dynamic queues only. You do this using the IMGRCOVQ queue attribute.

See ALTER QUEUES for more information on the IMGRCOVQ attribute.

Notes:

Media images are supported only if you are using linear logging. If you have enabled automatic media images, but are using circular logging, an error message is issued and the automatic media images attribute of the queue manager is disabled.
If you have enabled automatic media images, but have not specified a frequency, either minutes or megabytes of log, an error message is issued and no automatic media images are written.
You can manually record a media image using rcdmqimg when you have set IMGSCHED(AUTO), if you want.
This enables you to take media images at a time that is suitable for your enterprise, for example, when your system is quiet. Automatic media imaging takes account of these manual media images, because taking a manual media image resets the interval and log length, before which the next automatic media image is taken.
In IBM MQ 9.0.2, the queue manager writes persistent messages only in media images, not nonpersistent messages. This can reduce the size of media images when migrating to IBM MQ 9.0.2 or later

Deciding how to set IMGLOGLN and IMGINTVL

Make IMGLOGLN and IMGINTVL large enough, so the queue manager is only spending a fraction of its time recording media images, but small enough so that:

Damaged objects can be recovered in a reasonable amount of time, and
Small enough so that your log fits on your disk without running out of space.

If you set IMGLOGLN, a good practice is to make IMGLOGLN many times the amount of data on your queues and many times the data rate of your workload. The larger you make IMGLOGLN, the less time your queue manager spends recording media images.

Similarly, if you set IMGINTVL, a good practice is to make IMGINTVL many times the amount of time the queue manager takes to record a media image. You can find out how long it takes to record a media image by recording one manually.

If you make IMGLOGLN and IMGINTVL too large, recovering a damaged object might take a very long time, because all the extents since the last media image have to be replayed.

Make IMGLOGLN and IMGINTVL small enough, so that the maximum time taken to recover a damaged object is acceptable to you.

Making IMGLOGLN and IMGINTVL very large, means that the log grows very large because media images are recorded so rarely.

Attention: Ensure that a log of this size fits comfortably on your log filesystem, as your workload will get backed out if the log filesystem fills completely.

You can set both IMGINTVL and IMGLOGLN. This might be useful to ensure that automatic media images are taken regularly during heavy workload (controlled by IMGLOGLN), but are still taken occasionally when the workload is very light (controlled by IMGINTVL).

IMGINTVL and IMGLOGLN are targets for the interval and log data length between which automatic media images are taken.

These attributes should not be seen as a fixed maximum or minimum. In fact, the queue manager might decide to schedule an automatic media image sooner, if the queue manager perceives that it is a really good time:

Because the queue is empty, so taking the media image is the most efficient in terms of performance, and
A media image has not been recorded for a while

On occasion the gap between automatic media images might be somewhat longer than either, or both, of IMGINTVL and IMGLOGLN.

The gap between media images might be larger than IMGLOGLN if the amount of data on queues is approaching IMGLOGLN. The gap between media images might be larger than IMGINTVL if it takes almost as long as IMGINTVL to record a media image.

This is poor practice because the queue manager would be spending much of its time recording media images.

When using automatic media image recording, the queue manager records a media image for each object and queue individually, so the queue manager tracks the interval and log length between images separately for each object.

Gradually over time, recording of media images becomes staggered, instead of recording media images for all objects at the same time. This staggering spreads out the performance impact of recording media images, and is another advantage of using automatic recording of media images over manual recording.

Taking media images manually - linear logging only

Recording a media image of a queue involves writing all the persistent messages from that queue to the log. For queues containing large volumes of message data, this involves writing a large amount of data to the log, and this process can impact the performance of the system while it is happening.

Recording media images of other objects is likely to be comparatively quick, since the media image of other objects does not contain user data.

You need to carefully consider when to record the media images of queues, so that the process does not interfere with your peak workload.

You must record the media image of all objects regularly, in order to update the oldest log extent needed for media recovery.

A good time to record the media image of a queue is when it is empty, because at that point no message data is written to the log. Conversely, a bad time is when the queue is very deep or has very large messages on it.

A good time to record the media image of a queue is when your system is quiet; whereas a bad time is during peak workload. If your workload is always quiet at midnight, for example, you might decide to record media images at midnight every night.

Staggering the recording of each of your queues can spread the performance impact out, and so lessen its effect. The longer it has been since you last recorded media images, the more important it becomes to record them, as the number of log extents required for media recovery is increasing.

Note: When performing media recovery, all the required log files must be available in the log file directory at the same time. Make sure that you take regular media images of any objects you might want to recover to avoid running out of disk space to hold all the required log files.

For example, to take a media image of all your objects in your queue manager, run the rcdmqimg command as shown in the following examples:

On Windows


rcdmqimg -m QMNAME -t all *

On UNIX and Linux®


rcdmqimg -m QMNAME -t all "*"

Running rcdmqimg moves the media log sequence number (LSN) forwards. For further details on log sequence numbers, see Dumping the contents of the log using the dmpmqlog command. rcdmqimg does not run automatically, therefore must be run manually or from an automatic task you have created. For more information about this command, see rcdmqimg and dmpmqlog.

Note: Messages AMQ7467 and AMQ7468 can also be issued at the time of running the rcdmqimg command.

Partial media images

It is good practice to use IBM MQ messages only for data that is expected to be consumed in the near future, so that each message is on a queue for a relatively short amount of time.

Conversely, it is poor practice to use IBM MQ messages to store data long term like a database.

It is also good practice to ensure your queues are relatively shallow, and poor practice to have deep queues whose messages have been on the queue for a long time.

By following these guidelines, you enables the queue manager to optimize the performance of automatic recording of media images.

Recording the media image of an empty queue is very efficient (from a performance point of view) whereas taking the media image of a queue with a large amount of data on it is very inefficient, because all that data has to be written to the log in the media image.

For shallow queues with recently put messages on it, the queue manager can make a further optimization.

If all the messages currently on the queue were put in the recent past, the queue manager might be able to record the media image on behalf of a time (recovery point) just before all the messages were put, and so be able to record the image of the empty queue. This process is very low cost in terms of performance.

If all the messages that were on the queue at the recovery point have subsequently been got, these messages do not need to be recorded in the media image, as they are no longer on the queue.

This is called a partial media image. Then, in the unlikely event that the queue needs to be recovered, all logs records that relate to this queue since the last media image, will be replayed, so restoring all the recently put messages.

Even if there were a few messages on the queue at the recovery point, that are currently on the queue (and so have to be recorded in the partial media image) it is still more efficient to record this smaller partial media image, than a full media image of all messages.

Ensuring that messages remain on queues for a brief amount of time is likely to improve the performance of automatic recording of media images.