mmmsgqueue command
Manages the IBM Spectrum Scale message queue.
Synopsis
mmmsgqueue enable { -N NodeName[,NodeName...] | NodeFile | NodeClass } [-q]or
mmmsgqueue disable [-q]or
mmmsgqueue status [-v] { -N NodeName[,NodeName...] | NodeFile | NodeClass } [-Y] [-q]or
mmmsgqueue list { --topics | --servers} [-q]or
mmmsgqueue config --add-nodes -N { NodeName[,NodeName...] | NodeFile | NodeClass } [-q]or
mmmsgqueue config { --remove | --upgrade-auth } [-q]or
mmmsgqueue config --wf-upgrade [-q]or
mmmsgqueue config --update-broker-settings [-q]Availability
Available with IBM Spectrum
Scale Advanced Edition,
IBM Spectrum
Scale Data Management Edition, 
IBM Spectrum
Scale Developer Edition, 
or IBM Spectrum
Scale Erasure Code Edition. Available on Linux x86 and Linux PPC
LE.
Description
The mmmsgqueue command is used to manage the IBM Spectrum Scale message queue. The queue is based on the Kafka open source queue and is contained in the Kafka RPMs. The message queue is used by the file audit logging function to track events. Error and informational messages are written to the /var/adm/ras/mmmsgqueue.log file on the node from which the command is issued and any nodes where sub-commands are executed.
Parameters
- mmmsgqueue enable
- Enables the IBM Spectrum Scale message queue. The message queue ZooKeeper daemons are started on the nodes defined by the kafkaZookeeperServers node class. This defaults to Linux nodes in the cluster that are defined as quorum nodes that have the message queue packages installed. There must be at least three Linux quorum nodes with the message queue packages installed. The message queue broker daemons will be started on the nodes defined by the kafkaBrokerServers node class. This defaults to the nodes defined by the cesNodes node class. There must be at least three nodes defined. If your environment does not have the cesNodes node class defined or it has fewer than three nodes defined, you can specify the nodes to start the broker daemons by using the -N option. The -N option supports a comma-separated list of nodes, a full path name to a file containing node names, or a predefined node class. No matter how they are defined, you must specify a minimum of three nodes as message queue servers/brokers nodes. These nodes must be Linux-based and have the message queue packages installed on them. There are four callbacks defined to manage the message queue daemons.
- mmmsgqueue disable
- Disables the IBM Spectrum Scale message queue. All file systems must be disabled for file audit logging. Stops any broker or ZooKeeper daemons that might be running. Deletes message queue callbacks. Node classes and configuration files remain defined. To completely remove all configuration data, issue mmmsgqueue config --remove.
- mmmsgqueue status
- Provides the status of all broker and ZooKeeper daemons and shows which nodes contain which daemons. The -v (verbose) option provides additional status for the broker service, ZooKeeper service, invalid brokers that are registered, and good brokers that are not registered. The -Y option displays the command output in a parseable format with a colon (:) as a field delimiter. Each column is described by a header.
- mmmsqueue list { --topics | --servers }
- Lists all message queue topics currently configured or a comma-separated list of the broker nodes with their respective ports. You should see a topic defined for every file system that is enabled for file audit logging. The message queue uses an internal topic called __consumer_offsets.
- mmmsgqueue config --add-nodes
- Adds one or more broker nodes to the existing message queue if the nodes are already in the cluster. The -N option is required and can be followed by a comma-separated list of nodes, a full path name to a file containing node names, or a predefined node class. Several checks are performed before the nodes are allowed to be added. One of the checks is ensuring that there is enough local disk space on the node to accommodate all of the file systems that are enabled for file audit logging. Additionally, any existing broker data is removed from the nodes to ensure that there is no corruption when the broker starts. After starting the broker service on the added nodes, the existing partitions that are associated with file audit logging are rebalanced to better distribute the workload to include the additional nodes. Finally, the consumer node class is adjusted to include the additional nodes and the required consumer processes are started on the additional nodes.
- mmmsgqueue config { --remove | --upgrade-auth }
- The --remove option disables the message queue if enabled. All file systems must be disabled for file audit logging. It removes the node classes and message queue configuration files that were created to support the message queue. The --upgrade-auth option is used to automate the file audit logging authentication upgrade.
- mmmsgqueue config --wf-upgrade
- Puts the ACLs in place that are needed for clustered watch
folder.
Registers callbacks that are needed for clustered watch
folder. Note: Clustered watch folder must be enabled before this option can work.
- mmmsgqueue config --update-broker-settings
- This option performs a rolling restart of all broker processes, which allows the inter-broker protocol version and broker log format version to be updated. This option should only be used after successfully upgrading IBM Spectrum Scale code and performing mmchconfig release=LATEST.
- -q
- Suppresses all [I] informational messages.
Exit status
- 0
- Successful completion.
- nonzero
- A failure has occurred.
Security
You must have root authority to run the mmmsgqueue command.
The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages.
Examples
- To enable the message queue on a cluster that does not have the cesNodes
node class defined or has fewer than three nodes defined, issue the following
command:
# mmmsgqueue enable -N hs22n55,hs22n56,c6f2bc3n10,c6f2bc3n2 [I] The kafkaZookeeperServers node class was successfully created with 3 member nodes. [I] The kafkaBrokerServers node class was successfully created with 4 member nodes. [I] Successfully created Kafka broker configuration file and added to CCR. [I] Successfully created Kafka ZooKeeper configuration file and added to CCR. [I] Enabling MsgQueue daemons. [I] Creating callbacks to control starting and stopping the MsgQueue daemons. [I] MsgQueue successfully enabled. - To disable the message queue, issue the following
command:
# mmmsgqueue disable [I] Disabling MsgQueue daemons. [I] Removing callbacks that control starting and stopping the MsgQueue daemons. [I] MsgQueue successfully disabled. - To display the current status of the message queue daemons, issue the following
command:
# mmmsgqueue status Node Contains Broker Contains ZooKeeper Name Broker Status ZooKeeper Status c6f2bc3n10.gpfs.net yes good yes good c6f2bc3n2.gpfs.net yes good no hs22n55.gpfs.net yes good yes good hs22n56.gpfs.net yes good yes good - To display a more verbose status for the message queue daemons, such
as broker and ZooKeeper service status, issue the following
command:
# mmmsgqueue status -v -N ibmnode1 Node Name : ibmnode1 Broker Node : yes Broker Service Status : good Broker Service Uptime : 23min Zookeeper Node : yes Zookeeper Service Status : good Zookeeper Service Uptime : 23min Zookeeper Registered Brokers : Good (4/4) Invalid Brokers Registered : 0 Good Brokers Not Registered : 0 - The following output is an example of issuing mmmsgqueue
status
-Y:
[root@ibmnode1 mmfs] 2018-10-11 13:05:09# mmmsgqueue status -Y mmmsgqueue::HEADER:version:RESERVED:RESERVED:RESERVED:nodeName:isBroker:brokerStatus:isZookeeper:zookeeperStatus: mmmsgqueue:::1::::ibmnode1.ss.net:yes:good:yes:good: mmmsgqueue:::1::::ibmnode2.ss.net:yes:good:yes:good: mmmsgqueue:::1::::ibmnode7.ss.net:yes:good:yes:good: [root@ibmnode1 mmfs] 2018-10-11 13:05:29# - The following output is an example of issuing mmmsgqueue
status -v -Y
(verbose):
[root@ibmnode1 mmfs] 2018-10-11 13:05:29# mmmsgqueue status -v -Y mmmsgqueue::HEADER:version:RESERVED:RESERVED:RESERVED:nodeName:fieldName:data:RESERVED: mmmsgqueue:::1::::ibmnode1.ss.net:isBroker:yes:: mmmsgqueue:::1::::ibmnode1.ss.net:brokerStatus:good:: mmmsgqueue:::1::::ibmnode1.ss.net:brokerUptime:5 days :: mmmsgqueue:::1::::ibmnode1.ss.net:isZookeeper:yes:: mmmsgqueue:::1::::ibmnode1.ss.net:zookeeperStatus:good:: mmmsgqueue:::1::::ibmnode1.ss.net:zookeeperUptime:5 days :: mmmsgqueue:::1::::ibmnode1.ss.net:zookeeperRegisteredBrokersStatus:Good (3/3):: mmmsgqueue:::1::::ibmnode1.ss.net:zookeeperInvalidRegBrokers:0:: mmmsgqueue:::1::::ibmnode1.ss.net:zookeeperGoodUnregBrokers:0:: mmmsgqueue:::1::::ibmnode2.ss.net:isBroker:yes:: mmmsgqueue:::1::::ibmnode2.ss.net:brokerStatus:good:: mmmsgqueue:::1::::ibmnode2.ss.net:brokerUptime:5 days :: mmmsgqueue:::1::::ibmnode2.ss.net:isZookeeper:yes:: mmmsgqueue:::1::::ibmnode2.ss.net:zookeeperStatus:good:: mmmsgqueue:::1::::ibmnode2.ss.net:zookeeperUptime:5 days :: mmmsgqueue:::1::::ibmnode2.ss.net:zookeeperRegisteredBrokersStatus:Good (3/3):: mmmsgqueue:::1::::ibmnode2.ss.net:zookeeperInvalidRegBrokers:0:: mmmsgqueue:::1::::ibmnode2.ss.net:zookeeperGoodUnregBrokers:0:: mmmsgqueue:::1::::ibmnode7.ss.net:isBroker:yes:: mmmsgqueue:::1::::ibmnode7.ss.net:brokerStatus:good:: mmmsgqueue:::1::::ibmnode7.ss.net:brokerUptime:5 days :: mmmsgqueue:::1::::ibmnode7.ss.net:isZookeeper:yes:: mmmsgqueue:::1::::ibmnode7.ss.net:zookeeperStatus:good:: mmmsgqueue:::1::::ibmnode7.ss.net:zookeeperUptime:5 days :: mmmsgqueue:::1::::ibmnode7.ss.net:zookeeperRegisteredBrokersStatus:Good (3/3):: mmmsgqueue:::1::::ibmnode7.ss.net:zookeeperInvalidRegBrokers:0:: mmmsgqueue:::1::::ibmnode7.ss.net:zookeeperGoodUnregBrokers:0:: [root@ibmnode1 mmfs] 2018-10-11 13:06:03# - To display the defined message queue topics, issue the following
command:
# mmmsgqueue list --topics 154_6372129557625143312_audit __consumer_offsets [I] Successfully completed running topic command: list. - To add nodes that are already in the cluster to the message queue,
issue the following
command:
# mmmsgqueue config --add-nodes -N node-vm5 [I] Verifying new MsgQueue nodes meet minimum local space requirements for File Audit Logging to be enabled on them. Depending on the number of nodes being added, this may take some time. [I] The new MsgQueue nodes meet minimum local space requirements for File Audit Logging to be enabled on them. [I] Removing possible previous MsgQueue data files from the new MsgQueue nodes. [I] Successfully removed possible previous MsgQueue data files from the new MsgQueue nodes. [I] The kafkaBrokerServers node class was successfully updated with 1 additional member nodes. [I] Adding additional partitions to one or more topics due to additional broker nodes. [I] Successfully completed adding additional partitions to one or more topics due to additional broker nodes. [I] Attempting to redistribute partitions for one or more topics due to additional broker nodes. [I] Starting consumer processes on additional broker nodes. Depending on how many nodes were added, this may take some time. [I] The kafkaAuditConsumerServers node class was successfully updated with additional nodes. [I] Successfully completed starting consumer processes on additional broker nodes. [I] Successfully completed adding additional broker nodes. - To disable and remove message queue node classes, callbacks, and configuration files, issue the
following
command:
# mmmsgqueue config --remove [I] Attempting to disable the MsgQueue. This may take some time. [I] Disabling MsgQueue daemons. [I] Removing callbacks that control starting and stopping the MsgQueue daemons. [I] MsgQueue successfully disabled. [I] Removing MsgQueue node classes and configuration information. [I] MsgQueue successfully disabled and configuration removed.
See also
Location
/usr/lpp/mmfs/bin