IBM MQ failsafe mode

The appliance enters IBM® MQ failsafe mode if it cannot fully access RAID storage when it starts.

The appliance requires access to file systems on the RAID storage. If these file systems are not available when the appliance starts then important services cannot be started. These services include high availability (HA), disaster recovery (DR), the IBM MQ Console, the IBM MQ REST API and queue managers. If a required file system is not available then the appliance firmware pauses start-up of the IBM MQ subsystem and it enters IBM MQ failsafe mode. In this mode general system administration for the appliance can be performed, but IBM MQ is not available. After the problem is resolved, the IBM MQ services either start automatically, or the appliance can be restarted to restore normal operation.

If a required file system is not available at start up then the following message is output in the log:
AMQ3527S: A required IBM MQ Appliance file system is unavailable.

Additional messages might also be output to the file MQSystem.log in the mqtemporary: file system, or by a log target that subscribes to the qmgr logging category. You can view the MQSystem.log file by using the web UI or you can use the show file command in the CLI.

mqa(config)# show file mqtemporary:///MQSystem.log

10/01/21 10:01:20 root mqdpinit 16996.1 AMQ6676I: The internal disks are being configured for use by MQ. Phase 1 of 5.
10/01/21 10:01:20 root mqdpinit 16996.1 AMQ6676I: The internal disks are being configured for use by MQ. Phase 3 of 5.
10/01/21 10:01:20 root mqdpinit 16996.1 AMQ6676I: The internal disks are being configured for use by MQ. Phase 4 of 5.
10/01/21 10:01:20 root mqdpinit 16996.1 AMQ3527S: A required IBM MQ Appliance file system is unavailable.

To diagnose and resolve the problem you log in to the command line interface (CLI) by using either SSH or the serial console.

First, confirm that the RAID storage is functioning correctly by using the commands show raid-physical-drive, show raid-logical-drive, and show raid-ssd.

If the RAID storage is operating correctly then enter the IBM MQ CLI command shell to determine which file system(s) are not available.

mqa(config)# mqcli
AMQ6680E: A required IBM MQ Appliance file system is unavailable.

Use the status command to identify which file system is unavailable. An
encrypted file system might be unavailable if its passphrase is not stored or
its stored passphrase is not valid. The dspfspass command can be used to view
the status of stored passphrases and the setfspass command can be used to store
a passphrase that is required.

A transient failure to mount a file system might be resolved by rebooting the
appliance. If the problem persists consult the error logs for more information
and contact your IBM support center if you require assistance.

The MQ CLI operates in failsafe mode until the required file system is
available and the CLI is then restarted.

mqa(mqcli-failsafe)# 

The command prompt is mqcli-failsafe to indicate that IBM MQ failsafe mode is active.

In IBM MQ failsafe mode the options that are available in the IBM MQ CLI command shell are restricted.

You use the status command to determine which file systems are unavailable.

You might enter IBM MQ failsafe mode because the IBM MQ system file systems are encrypted and their passphrase is not stored on the internal flash. This situation occurs if you have cleared a file system passphrase before restarting the appliance, or you have moved the SSD storage to a replacement appliance following a hardware failure.

mqa(mqcli-failsafe)# status
Memory:                                  187.8GB total, 3383MB used [2%]
CPU:                                     0%
CPU load:                                0.14, 0.87, 0.57
Internal disk:                           2979.9GB total, 196608MB used [6%]
System volume:                           29.4GB total, 6629MB used [22%]
MQ root file system:                     0.0GB total, 0MB used [100%] [encrypted, not available, passphrase not stored]
MQ diagnostics file system:              0.0GB total, 0MB used [100%] [encrypted, blocked, passphrase not stored]
MQ errors file system:                   0.0GB total, 0MB used [100%] [encrypted, blocked, passphrase not stored], 0 FDCs
MQ trace file system:                    0.0GB total, 0MB used [100%] [encrypted, blocked, passphrase not stored]
You must provide the file system passphrases to the replacement appliance by using the setfspass command. For example, to store the passphrase for the IBM MQ root file system you can use the following command, where the passphrase is specified by using the -p parameter (you can alternatively omit the -p parameter and specify the passphrase interactively):
setfspass -f root -s -p red-white-blue

Each file system is mounted automatically when its passphrase is stored unless a file system that it depends on is unavailable. If a file system that it depends on is unavailable then the status command reports the file system is blocked.

For example, after providing the passphrase for the root and the diagnostics file systems, you might see the following status:
mqa(mqcli-failsafe)# status
Memory:                                  187.8GB total, 3422MB used [2%]
CPU:                                     0%
CPU load:                                0.09, 0.08, 0.07
Internal disk:                           2979.9GB total, 196608MB used [6%]
System volume:                           29.4GB total, 6629MB used [22%]
MQ root file system:                     15.8GB total, 175MB used [1%] [encrypted]
MQ diagnostics file system:              63.0GB total, 196MB used [0%] [encrypted]
MQ errors file system:                   0.0GB total, 0MB used [100%] [encrypted, not available, passphrase not stored], 0 FDCs
MQ trace file system:                    0.0GB total, 0MB used [100%] [encrypted, not available, passphrase not stored]

When all the required file systems are available the firmware will resume start-up for the IBM MQ subsystem.

mqa(mqcli-failsafe)# setfspass -f trace -s -p red-white-blue
File system passphrase stored.

The required IBM MQ Appliance file systems are now available.
The IBM MQ Appliance subsystem is starting.

To confirm the appliance is operating normally, exit the mqcli-failsafe shell, then re-enter mqcli.

You can then enter the passphrase for other encrypted file systems, if required.

Note: You might also enter IBM MQ failsafe mode if you scheduled a file system to be encrypted or decrypted when the appliance restarts, and you then access the IBM MQ CLI during start-up before the scheduled action completes. The status command can be used to monitor the progress of the scheduled action. On completion, the file system is mounted and the appliance continues start-up automatically.

If you cannot diagnose or resolve the cause of IBM MQ failsafe mode then contact IBM Support for assistance.