Recovering WebSphere MQ and WebSphere Message Broker after a server crash


This article is about disaster recovery, specifically recovering the queue manager and broker after a server crash involving IBM® WebSphere® MQ V7 and IBM WebSphere Message Broker V7/V8. A server crash can consist of any abrupt restart of the server that does not allow WebSphere MQ and WebSphere Message Broker shutdown routines to complete before the server goes down. It includes server hangs where the administrator must reboot the server, and the shutdown routines are not able to complete. This article describes error scenarios following abrupt server restart, and shows system administrators how to recover from such situations.

Existing resources such as information centers provide standard WebSphere MQ and WebSphere Message Broker restore commands, but lack step-by-step troubleshooting information and additional steps that are sometimes needed to recover the environment. This article is for WebSphere MQ and WebSphere Message Broker administrators, and Integration and Infrastructure Architects who need to guide their Support Teams in recovering from a server crash. Many customers have business-critical applications that use WebSphere MQ as a messaging system and WebSphere Message Broker for transformation, enrichment, and routing, so detailed information on this topic is worth knowing.

Recovering the MQ queue manager


You attempt to restart the MQ queue manager after a server crash and startup fails.


The queue manager active logs appear okay, and the command strmqm –c for recovering any damaged system objects does not work. When starting the queue manager, you may see this error:

WebSphere MQ queue manager 'qmgr name' starting.
AMQ7047: An unexpected error was encountered by a command.

/var/mqm/qmgrs/QM Name/errors shows up the below error

05/24/13 19:25:17 - Process(6160464.1) User(mqm) Program(amqzxma0_nd)
AMQ7472: Object qmgr name, type catalogue damaged.

EXPLANATION:  Object qmgr name, type catalogue has been marked as damaged. 
This indicates that the queue manager was either unable to access the object in the file
system, or that some kind of inconsistency with the data in the object was detected.

ACTION: If a damaged object is detected, the action performed depends on whether the 
queue manager supports media recovery and when the damage was detected. If the queue
 manager does not support media recovery, you must delete the object as no recovery is 
possible. If the queue manager does support media recovery and the damage is detected 
during processing when the queue manager is being started, the queue manager 
automatically initiates media recovery of the object. If the queue manager supports media
recovery and the damage is detected after the queue manager has started, it may be 
recovered from a media image using the rcrmqobj command or it may be deleted.

You may also see some "ghost" MQ objects in the folder /var/mqm/qmgrs/QM Name/queues, as shown below:


An FDC with the following probe ID and major and minor error codes is generated:

Probe Id:         AO000001
Component:        apiStartup
Major Errorcode:  xecF_E_UNEXPECTED_RC
Minor Errorcode:  arcE_OBJECT_DAMAGED
Probe Type:       MSGAMQ6118


The abrupt server restart has corrupted the MQ object catalogue.

Queue manager object catalog

The object catalog contains the details of all WebSphere MQ objects contained in the queue manager. It is used during queue manager startup, and a corrupted object catalog will prevent startup. The object catalog is located at /var/mqm/qmgrs/qmgr Name/qmanager.

Resolving the problem

Startup of the queue manager automatically recovers from an object catalogue in such a state without reporting an FDC, but only if the queue manager has media recovery enabled -- that is, only if you are using linear logging. If you are using circular logging, then no recovery is possible, and you have two options to recover the queue manager:

  1. Restore the queue manager configuration from the file system backup at /var/mqm/qmgrs/qmgr name.
  2. Delete the queue manager, re-create it, and restore the configuration from the saveqmgr backup.

Option 1 is preferable because a working copy of the queue manager object catalog is all that you need to recover the queue manager, and you don't have to deal with the complexities of a QMID change which is required with Option 2. Here are the steps to recover the queue manager from file system backup:

  1. Ask your media tape backup team to restore the /var/mqm/qmgrs/qmgr name file system with the backup taken couple of days before the server crash.
  2. After the file system is restored, try starting the queue manager. If you see the same error, repeat the above step with progressively older backups and try to start the queue manager again.
  3. If you are still unable to find a working backup, you will need to use Option 2 above.

Here are the steps to recover the queue manager from saveqmgr backup:

  1. Delete the corrupted queue manager: dltmqm qmgr name. Be sure to make a backup copy of qm.ini -- you will need it when re-creating the queue manager to determine the number of primary and secondary log files, log file pages, and any additional custom settings on the current queue manager.
  2. Re-create the queue manager using the command crtmqm qmgr name. You can add the –lf switch to specify the log file size and –lp and –ls to specify the number of primary and secondary log files to match up with the old queue manager.
  3. Update the qm.ini file of the new queue manager with any additional settings you backed up in the old qm.ini file.
  4. Start the queue manager: strmqm qmgr name.
  5. Restore the queue manager configuration: runmqsc qmgr name < backup file name. For example, if your queue manger is QM1 and the backup file is QM1.mqsc, then run the command: runmqsc QM1 <QM1.mqsc:
    /var/mqm > runmqsc  <QM1.MQSC
    No commands have a syntax error. 
    All valid MQSC commands were processed.
  6. Apply the authorities: ./qmgr authorities backup.oam
  7. For example, if your queue manger authorities backup file is QM.oam, then execute it using ./ QM1.oam:
    /var/mqm > runmqsc  ./QM1.AUT
    The setmqaut command completed successfully.
  8. Now that the queue manager is restored, check the queue manager cluster configuration using the command: DIS CLUSQMGR (qmgr name) CLUSTER (cluster name).
  9. For QMGR QM1 and cluster CLUS1, run the command: DIS CLUSQMGR(QM1) CLUSTER(CLUS1).
  10. You will see two entries for QM1 with a two different QMIDs in the cluster. Determine which QMID corresponds to the old queue manager and FORCEREMOVE it from the cluster with the command: RESET CLUSTER(cluster name) QMID(qmid) ACTION(FORCEREMOVE). You can FORCEREMOVE a queue manager from a cluster only from a full repository queue manager. The queue manager should now be in a healthy state.

Recovering the broker


After starting up the queue manager, you start the broker and, and while it starts successfully, it is non-responsive. Even an mqsilist command takes a long time to show output, and sometimes it does not show any output at all. Because the queue manger has been re-created and the QMID has changed, the broker also needs to be re-created, and its configuration needs to be restored from the backup created with the mqsibackupbroker command. Sometimes the problem may persist even after you re-create the broker.


Execution groups may not bind to their corresponding queues, applications may fail to connect to SOAP ports, and broker deployments may fail. WebSphere MQ logs may show the following error:

05/27/13 04:37:20 - Process(9044138.1) User(mqm) Program(amqzxma0_nd) 
AMQ7159: A FASTPATH application has ended unexpectedly.

EXPLANATION: A FASTPATH application has ended in a way that did not let the queue manager
clean up the resources owned by that application. Any resources held by the application 
can be released only by stopping and restarting the queue manager.

ACTION: Investigate why the application ended unexpectedly. Avoid ending FASTPATH 
applications in a way that prevents WebSphere MQ from releasing resources held by the 

WebSphere Message Broker logs will show the following errors for most or all of the execution groups in the broker logs:

May 27 04:24:25 hostname user:err|error WebSphere Broker 
v7005[18415870]: (Broker Name.EG Name)[1]BIP2116E: 
Message broker internal error: diagnostic information 'Fatal Error; exception thrown 
before initialisation completed', 'Load LILs', '18415870', '1', '13', '12'. :
Broker Name.e404e800-3401-0000-0080-af98194179f4:
/build/S700_P/src/DataFlowEngine/ImbMain.cpp: 252:
ImbMain::ProgressChecker::~ProgressChecker: :

May 27 04:24:25 hostname user:info WebSphere Broker v7005[21692514]:
(Broker Name.EG Name)[1]BIP7050E: 
Failed to locate Java class
(Broker Name..2ecf3aaf-3101-0000-0080-b2a00806016c:
/build/S700_P/src/DataFlowEngine/NativeTrace/ImbNativeTrace.cpp: 698: getClasses: :

Possible causes

  • Broker non-responsiveness may be caused by the change in the QMID of the broker queue manager, or by the corruption of the broker due to the abrupt server restart. Therefore the first option is to delete and re-create the broker and restore the configuration from the backup created with the mqsibackupbroker command.
  • Another possible cause is broker overload, which you can investigate by creating a test broker and then creating a sample execution group. If the broker still does not respond to mqsi commands, then the broker install image is probably corrupted.
  • An abrupt server restart can corrupt the broker install image in a way that causes the Java libraries to fail to load. The broker administrator needs to determine whether the corruption of the broker is limited to just the latest fix pack, or whether it involves the entire broker install image.

Resolving the problem

Follow the steps below to re-create the broker:

  1. Stop the broker using the command mqsistop broker name.
  2. Delete the broker using the command below. Do not specify option –s with the mqsideletebroker command, because it will delete all broker administrative security queues (SYSTEM.BROKER.AUTH and SYSTEM.BROKER.AUTH.egname) from the queue manager. These queues will be reused when the broker is re-created using mqsideletebroker broker name.
  3. Create the broker using the command mqsicreatebroker broker name -q qmgr name.
  4. Start the broker to verify that it starts up okay. Check the broker logs for any errors using mqsistart broker name.
  5. Stop the broker again and issue the restore broker command
    mqsirestorebroker broker name -d directory path -a backup archive name:
    mqsi restorebroker BKR1 -d /var/mqsi/backup -a
    BIP1266I: Deleting existing configuration of broker BRK1
    BIP1268I: Recreating broker configuration information from backup archive 
    BIP8071I: Successful command completion
    You have mail in /usr/spool/mail/mbadm
  6. Start the broker. The QMID change is no longer the issue, but the broker may be still unresponsive.
  7. To ascertain that the issue is not with broker overload, create a test broker and then a test execution group in the broker. The broker gets created, but the execution group creation will time out. If you create an execution group with the –w option, the execution group may get created, but response to mqsi commands will take a long time, which shows that the issue is not with broker overload.
  8. Now that you know that the broker non-responsiveness is caused by corruption in the broker install image. Identify the level of fix pack applied. On distributed platforms, there is no option to selectively reinstall the fix pack, so you will have to uninstall the broker component and then reinstall it at the appropriate service level. For details on uninstalling and reinstalling WebSphere Message Broker, see either:
  9. After you reinstall the broker, start it and run the mqsi commands. Broker response should be normal.


This article described recovery procedures for WebSphere MQ and WebSphere Message Broker after a server crash. It showed you restore procedures for WebSphere MQ, both from the file system and from saveqmgr (which is included in WebSphere MQ V7.5 or later). It also covered restore procedures for WebSphere Message Broker, including troubleshooting tips to fix a non-responsive broker issue.


The authors would like to thank the IBM PMR support organization, especially S. Rao, for their technical assistance. Thanks also to Srinivas Aparadhi, IBM Technical Services Professional and Software Specialist, for his technical assistance.

Downloadable resources

Related topics


Sign in or register to add and subscribe to comments.

ArticleTitle=Recovering WebSphere MQ and WebSphere Message Broker after a server crash