Managing log files

Information specific to circular logging

If you are using circular logging, ensure that there is sufficient space to hold the log files when you configure your system (see LogDefaults stanza of the mqs.ini file and Log stanza of the qm.ini file). The amount of disk space used by the log does not increase beyond the configured size, including space for secondary files to be created when required.

Information specific to linear logging

If you are using a linear log, the log files are added continually as data is logged, and the amount of disk space used increases with time. If the rate of data being logged is high, disk space is used rapidly by new log files.

To determine "older" and "newer" log files, use the log file number rather than the modification times applied by the file system.

Information applicable to both types of logging

Only log files required for queue manager restart, active log files, are required to be online. Inactive log files can be copied to an archive medium such as tape for disaster recovery, and removed from the log directory. Inactive log files that are not required for media recovery can be considered as superfluous log files. You can delete superfluous log files if they are no longer of interest to your operation.

If any log file that is needed cannot be found, operator message AMQ6767E is issued. Make the log file, and all subsequent log files, available to the queue manager and try the operation again.

Cleaning log extents automatically - linear logging only

From IBM MQ 9.1.0 you have the option to use automatic management of linear log extents no longer required for recovery.

You use the LogManagement attribute in the Log stanza of the qm.ini file, or by using the IBM MQ Explorer, to set up automatic management. See Log stanza of the qm.ini file for more information.

Taking media images automatically - linear logging only

From IBM MQ 9.1.0 there is an overall switch to control whether the queue manager automatically writes media images, the default being that the switch has not been set.

If you have a critical time during the day when workload is very heavy and you want to be sure that system throughput is not impacted by taking automatic media images, you might wish to temporarily switch off automatic media imaging by setting IMGSCHED(MANUAL).

For internal system objects, such as the object catalog and the queue manager object, the queue manager automatically writes media images as appropriate.

See ALTER QMGR for more information on the attributes.

You can also enable or disable automatic and manual media images for local and permanent dynamic queues only. You do this using the IMGRCOVQ queue attribute.

See ALTER QUEUES for more information on the IMGRCOVQ attribute.

Deciding how to set IMGLOGLN and IMGINTVL

By default, IMGLOGLN is set to off for queue managers other than Native HA queue managers. (Native HA queue managers are created with IMGLOGLN set to the value of 25% of the available space on the volume where the recovery logs are to be written.)

By default, IMGINTVL is set to 60 minutes. The interval specified by IMGINTVL is honored when enough new work has been carried out on the queue manager to make it worthwhile recording a new image. Otherwise the taking of new images is delayed.

If you set IMGLOGLN, a good practice is to make IMGLOGLN many times the amount of data on your queues and many times the data rate of your workload. The larger you make IMGLOGLN, the less time your queue manager spends recording media images.

Similarly, if you set IMGINTVL, a good practice is to make IMGINTVL many times the amount of time the queue manager takes to record a media image. You can find out how long it takes to record a media image by recording one manually.

If you make IMGLOGLN and IMGINTVL too large, recovering a damaged object might take a very long time, because all the extents since the last media image have to be replayed.

Make IMGLOGLN and IMGINTVL small enough, so that the maximum time taken to recover a damaged object is acceptable to you.

You can set both IMGINTVL and IMGLOGLN. This might be useful to ensure that automatic media images are taken regularly during heavy workload (controlled by IMGLOGLN), but are still taken occasionally when the workload is very light (controlled by IMGINTVL).

IMGINTVL and IMGLOGLN are targets for the interval and log data length between which automatic media images are taken.

On occasion the gap between automatic media images might be somewhat longer than either, or both, of IMGINTVL and IMGLOGLN.

The gap between media images might be larger than IMGLOGLN if the amount of data on queues is approaching IMGLOGLN. The gap between media images might be larger than IMGINTVL if it takes almost as long as IMGINTVL to record a media image.

This is poor practice because the queue manager would be spending much of its time recording media images.

When using automatic media image recording, the queue manager records a media image for each object and queue individually, so the queue manager tracks the interval and log length between images separately for each object.

Gradually over time, recording of media images becomes staggered, instead of recording media images for all objects at the same time. This staggering spreads out the performance impact of recording media images, and is another advantage of using automatic recording of media images over manual recording.

Taking media images manually - linear logging only

Recording a media image of a queue involves writing all the persistent messages from that queue to the log. For queues containing large volumes of message data, this involves writing a large amount of data to the log, and this process can impact the performance of the system while it is happening.

Recording media images of other objects is likely to be comparatively quick, since the media image of other objects does not contain user data.

You need to carefully consider when to record the media images of queues, so that the process does not interfere with your peak workload.

You must record the media image of all objects regularly, in order to update the oldest log extent needed for media recovery.

A good time to record the media image of a queue is when it is empty, because at that point no message data is written to the log. Conversely, a bad time is when the queue is very deep or has very large messages on it.

A good time to record the media image of a queue is when your system is quiet; whereas a bad time is during peak workload. If your workload is always quiet at midnight, for example, you might decide to record media images at midnight every night.

Staggering the recording of each of your queues can spread the performance impact out, and so lessen its effect. The longer it has been since you last recorded media images, the more important it becomes to record them, as the number of log extents required for media recovery is increasing.

Partial media images

It is good practice to use IBM MQ messages only for data that is expected to be consumed in the near future, so that each message is on a queue for a relatively short amount of time.

Conversely, it is poor practice to use IBM MQ messages to store data long term like a database.

It is also good practice to ensure your queues are relatively shallow, and poor practice to have deep queues whose messages have been on the queue for a long time.

By following these guidelines, you enables the queue manager to optimize the performance of automatic recording of media images.

Recording the media image of an empty queue is very efficient (from a performance point of view) whereas taking the media image of a queue with a large amount of data on it is very inefficient, because all that data has to be written to the log in the media image.

For shallow queues with recently put messages on it, the queue manager can make a further optimization.

If all the messages currently on the queue were put in the recent past, the queue manager might be able to record the media image on behalf of a time (recovery point) just before all the messages were put, and so be able to record the image of the empty queue. This process is very low cost in terms of performance.

If all the messages that were on the queue at the recovery point have subsequently been got, these messages do not need to be recorded in the media image, as they are no longer on the queue.

This is called a partial media image. Then, in the unlikely event that the queue needs to be recovered, all logs records that relate to this queue since the last media image, will be replayed, so restoring all the recently put messages.

Even if there were a few messages on the queue at the recovery point, that are currently on the queue (and so have to be recorded in the partial media image) it is still more efficient to record this smaller partial media image, than a full media image of all messages.

Ensuring that messages remain on queues for a brief amount of time is likely to improve the performance of automatic recording of media images.