mmces command

Availability

Available on all IBM Spectrum Scale editions.

Description

Use the mmces command to manage protocol addresses, services, node state, logging level and load balancing. Assignment of the CES addresses to the network adapters is done by the CES base address specified in CES legacy mode, or by a user-specified configuration in the CES interface mode. The protocol functions provided in this command, or any similar command, are generally referred to as CES (Cluster Export Services). For example, protocol node and CES node are functionally equivalent terms. Before using the mmces command, CES has to be installed and configured in the system. For more information on how to install CES, see Installing Cluster Export Services as part of installing IBM Spectrum Scale on Linux systems.

CES currently supports the NFS, SMB, BLOCK, HDFS, and Object services. Each service can be enabled or disabled with the mmces service command. Enabling a service is a CES cluster-wide operation. In addition, enabled services can be started and stopped on individual nodes.

Clients access the CES services using one or more IP addresses in the CES address pool. Addresses can be added to and removed from the pool using the mmces address add and mmces address remove commands. Existing addresses can be reassigned to another node with the mmces address move command.

Addresses can have one or more attributes associated with them. An address attribute is a tag that the services can identify a specific address as having a special meaning, which is defined by the service protocol. Addresses can have multiple attributes, but an attribute can only be associated with a single address. The supported attributes are object_singleton_node, object_database_node, and HDFS-related attributes. The HDFS-related attributes are assigned automatically. These attributes are used to manage HDFS and object protocol-related services. For more information, see the Object services and object protocol nodes table in Understanding and managing Object services.

Addresses can have a policy associated with them, and that policy determines how addresses are automatically distributed. The allowed policies are none, balanced-load, node-affinity, and even-coverage. A policy of none means addresses are not distributed automatically.

The assignment of addresses to a Network Interface Controller (NIC) can be done by using one of these two modes:

• CES legacy mode

  An address with the same subnet mask as the CES addresses, called the CES Base Address, is defined on each CES node. CES addresses are assigned to a NIC where the CES Base Address is hosted. In this mode, only IPv4 addresses are supported and no other non-CES address with the same subnet mask as the CES Base Address should exist on CES nodes.

• CES interface mode
  The customer defines one or more individual NICs on each CES node. The CES addresses are assigned to these NICs. To manage subnets, like it is currently done for CES nodes, a NIC can be a member of one or more CES groups. In this case a CES address can only be assigned to a NIC if the NIC is in the same group as the CES address. In this mode IPv6 and IPv4 addresses are supported.

  Note: The address balancing done by policies works node-wide. This means that if a CES address can be hosted on different NICs on the same node, the NIC where the address is hosted is not defined. Well-defined address assignment on a node with multiple NICs is achieved by the usage of CES group.

Transitioning from legacy mode to interface mode: In a legacy mode, no IPv6 addresses can be added. You can switch from the legacy mode to the interface mode only if you have a valid NIC configuration. To switch from a legacy mode to an interface mode, add the NIC first and then switch the CES mode to interface. When the NICs are specified in the legacy mode you are in transition. This state is allowed but not recommended. You must switch to the interface mode, or remove the NIC configuration by using the mmces interface mode interface command.

Transitioning from interface mode to legacy mode: Use the mmces interface mode legacy --force command to switch back to the legacy mode while keeping the NIC assignment. The mmces interface mode legacy --force command deletes the NIC configuration and sets the cluster in legacy mode. Transitioning from the interface mode to the legacy mode unassigns all the IPv6 addresses. If you do not have CES Base Addresses, all the IPv4 addresses are also unassigned.

Use the mmces interface command to manage CES modes and NIC assignment.

A CES node can be placed in a suspended state. When a node is in suspended state, all of the CES addresses for that node are reassigned to other nodes, or set to unassigned. The node will not accept new address assignments. Any services that are started when the node is suspended continue to run, except mmcesmonitord. The mmcesmonitord service is stopped, and the lock on the sharedRoot file system is released. The sharedRoot file system is still mounted, but can be unmounted. The suspended state is persistent, which means nodes remain suspended following a reboot. After a reboot no services are running on a suspended node. However, if a node is not suspended, the services that were enabled on the nodes are restarted after the reboot.

The presence of a CES IP on a name node in an HDFS group activates the HDFS service. When there are multiple name nodes in an HDFS group, only the name node which has a CESIP assigned to it can be active. The absence of a CES IP deactivates the HDFS service from that name node. An HDFS group has the prefix hdfs followed by at least one alphabetical or numeric character. For example, hdfsbda1. You can assign a group or an attribute to a CES IP. If an HDFS group is assigned or unassigned from a CES IP, the related attribute is also assigned or unassigned automatically. A CES IP assigned to an HDFS group must not have any attributes other than the ones automatically assigned. An HDFS group must be assigned only once to all the CES IPs in the cluster.

Parameters

service

Manages protocol services with one of the following actions:

enable

Enables and starts the specified service on all CES nodes.

disable

Disables and stops the specified service on all CES nodes.

Note: Disabling a service will discard any configuration data from the CES cluster and needs to be used with caution. If applicable, backup any relevant configuration data. Subsequent service enablement will start with a clean configuration.

--force

Allows the service disable action to go through without confirmation from the user.

start

Starts the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is started on the local node.

stop

Stops the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is stopped on the local node.

Note: If a service is stopped on a node that has CES addresses assigned, clients will not be able to access the service using any of the addresses assigned to that node. Access to the data from clients is not possible any more for services that are stopped. This state is not persistent, so after a reboot all the services become active again.

list

Lists the state of the enabled services.

node

Manages CES node state with one of the following actions:

suspend

Suspends the specified nodes. If neither the -N or -a parameters are specified, only the local node is suspended. The -a stands for all CES nodes.

When a node is suspended, all addresses assigned to the node are reassigned to other nodes, or set to unassigned. The node will not accept any subsequent address assignments. Suspending a node triggers CES protocol recovery if the node has CES addresses assigned. This does not stop any services or the mmces monitor daemon.

If the --stop option is added to the suspend command, all the CES services including the mmces monitor daemon are stopped. This releases the lock on the sharedRoot file system, and it can be unmounted without further actions. The sharedRoot file system is still mounted, but can be unmounted.

Note: After a reboot no services are running on a suspended node. However, if a node is not suspended, the services that were enabled on the nodes are restarted after the reboot.

resume

Resumes the node so that CES IPs can be retrieved and starts the mmces monitor daemon if it is not running.

Using the --start option starts all the enabled protocols. The mmces monitor daemon is also started, if it is not already running. However, if at least one protocol was not successfully started, then all the protocols that were already running are also stopped.

list

Lists the specified nodes along with their current node state. If the -N parameter is not specified, all nodes are listed.

verbose

Lists the addresses assigned to the nodes.

--ces-group

Lists the nodes belonging to the specified groups.

log level

Sets the CES log level, which determines the amount of CES related information that is logged in the file, /var/adm/ras/mmfs.log.latest. The log level values can range from 0, which is the default value, to 3. Increasing the log level adds to the information being logged.

The values are defined as follows:

level 0

Logs only non-repeated errors and non-repeated warnings. The non-repeated warnings indicate faulty behavior. Information about the state of an action is also logged during startup and shutdown.

level 1

Logs all errors and all non-repeated warnings.

level 2

Logs all warnings.

level 3

Logs all important debug information.

Important: A higher log level logs the data of all the lower log levels. Therefore, log level 1 includes the information of log level 0, log level 2 includes the information of log level 1 and log level 0, and so on.

The following status information is also accumulated in the log:

error

When an event that degrades the functionality of the system has occurred.

warning

When an unexpected event has occurred. However, this event does not degrade the functionality of the system. For example, an error was solved by a retry.

information

Documents the result of an operation. This is logged during startup and shutdown or in log level 3.

address

Manages CES addresses with one of the following actions:

add

Adds the addresses specified by the --ces-ip parameter to the CES address pool and assigns them to a node. The node to which an address is assigned will configure its network interfaces to accept network communication destined for the address. CES addresses must be different from IP addresses used for GPFS or CNFS communication.

If --ces-node is specified with add, all addresses specified with the --ces-ip parameter will be assigned to this node. If --ces-node is not specified, the addresses will be distributed among the CES nodes.

If an attribute is specified with --attribute, there can only be one address specified with the --ces-ip parameter.

If --ces-group is specified with add, all new addresses will be associated with the specified group. The result can be viewed with the mmces address list command.

Note: The provided addresses or host names must be resolvable by forward and reverse name resolution (DNS or /etc/hosts on all CES nodes). Otherwise you get the following error message:

Cannot resolve <ip address>; Name or service not known

You can also perform a manual check by running the following command: mmcmi host <ip address>.

Ensure that the netmask (PREFIX) setting in the ifcfg-<interface> files is correct.

remove

Removes the addresses specified by the --ces-ip parameter from the CES address pool. The node to which the address is assigned reconfigures its network interfaces to no longer accept communication for that address.

move

Moves addresses.

If the --ces-ip parameter is specified, the addresses specified by IP are moved from one CES node to another. The addresses are reassigned to the node specified by the --ces-node parameter.

If the --rebalance parameter is specified, the addresses are distributed within 60 seconds based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.

Note: The information about the address movement is printed after the rebalance is done. The address movement is also done in the background periodically.

Use this command with caution because IP movement will trigger CES protocol recovery.

change

Changes or removes address attributes.

If the --ces-ip parameter is specified:

• The command associates the attributes that are specified by the --attribute parameter with the address that is specified by the --ces-ip parameter. If an attribute is already associated with another address, that association is ended.
• If the --remove-attribute parameter is specified, the command removes the attributes that are specified by the --attribute parameter from the addresses that are specified by the --ces-ip parameter.
• The command associates the groups that are specified by the --ces-group parameter with the address that is specified by the --ces-ip parameter
• If the --remove-group parameter is specified, the command removes the groups that are specified by the --ces-group parameter from the addresses that are specified by the --ces-ip parameter.

If the --ces-ip parameter is not specified:

• If the --remove-attribute parameter is specified, the command removes the attributes that are specified by the --attribute parameter from their current associations.
• If the --remove-group parameter is specified, the command removes the groups that are specified by the --group parameter from their current associations.
• Specifying --remove-group with the groups specified by the --group parameter removes the groups from their current associations.

list

Lists the CES addresses along with group, attribute and node assignments.

Options:

• --ces-ip List only the addresses provided.
• --ces-group List only addresses whose group assignment matches one of the groups provided.
• --attribute List only addresses whose attributes match one of the attributes provided.
• --by-node List addresses by node, using the output format from IBM Spectrum Scale V4.1.1 and later.
• --extended-list Lists the preferred node of the given address in a new column if the address balancing mode option is set to node-affinity.
• --full-list Lists the information about the preferred node and the node names where the given address could not be hosted in two new columns.

Note: The [-N {Node[,Node...]| NodeFile | NodeClass| -a] option is deprecated for IBM Spectrum Scale version 4.2.3 and might be removed in a later release.

policy

Sets the CES address distribution policy.

state

Shows the state of one or more nodes in the cluster.

show

Shows the state of the specified service on the nodes specified. If no service is specified, all services will be displayed. If neither the -N or -a parameters are specified, the state of the local node is shown.

cluster

Shows the combined state for the services across the whole CES cluster. If no service is specified an aggregated state will be displayed for each service, where healthy means the service is healthy on all nodes, degraded means the service is not healthy on one or all nodes, and failed means that the service is not available on any node. If a service is specified the state of that service will be listed for each node, along with the name of any event that is contributing to an unhealthy state.

--ces-node

Indicates that the command applies only to the specified CES node name.

--attribute

Specifies either a single attribute or a comma-separated list of attributes as indicated in the command syntax.

--ces-ip

Specifies either a single or comma-separated list of DNS qualified host names or IP addresses as indicated in the command syntax.

--rebalance

Distributes addresses immediately based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.

none

Specifies that addresses are not distributed automatically.

balanced-load

Distributes addresses dynamically in order to approach an optimized load distribution throughout the cluster. The network and CPU load on all the nodes is monitored and addresses are moved based on given policies.

Addresses that were recently moved or addresses with attributes are not moved.

node-affinity

Attempts to keep addresses associated with the node to which they were assigned. Address node associations are created with the --ces-node parameter of the mmces address add command or the mmces address move command. Automatic movements of addresses do not change the association. Addresses that were enabled without a node specification do not have a node association. Addresses that are associated with a node but assigned to a different node are moved back to the associated node.

Addresses that were recently moved or addresses with attributes are not moved.

even-coverage

Attempts to evenly distribute all of the addresses among the available nodes.

Addresses that were recently moved or addresses with attributes are not moved.

--remove-attribute

Indicates that the specified attributes should be removed.

-N {Node[,Node...] | NodeFile | NodeClass}

Indicates that the command applies only to the specified node names.

For general information on how to specify node names, see Specifying nodes as input to GPFS commands.

-a

Specifies that the command applies to all CES nodes.

NFS

Specifies that the command applies to the NFS service.

OBJ

Specifies that the command applies to the Object service.

SMB

Specifies that the command applies to the SMB service.

BLOCK

Specifies that the command applies to the BLOCK service.

AUTH

Specifies that the command applies to the AUTH service.

NETWORK

Specifies that the command applies to the NETWORK service.

CES

Specifies that the command applies to the CES service.

--verbose

Specifies that the output is verbose.

new-level

Sets the log level to a new value. If the new-level parameter is not specified, the current log level is displayed.

-Y

Displays the command output in a parseable format with a colon (:) as a field delimiter. Each column is described by a header.

Note: Fields that have a colon (:) are encoded to prevent confusion. For the set of characters that might be encoded, see the command documentation of mmclidecode. Use the mmclidecode command to decode the field.

--time

Lists the previous events from one of the following intervals:

hour

Lists the events from the past hour.

day

Lists the events from the past day.

week

Lists the events from the past week.

month

Lists the events from the past month.

The events are listed whether or not they are currently contributing to the state of a component.

interface

Shows the state of one or more nodes in the cluster.

mode

Manage the CES mode.

legacy

Set the cluster in CEC legacy mode. IPv6 support is disabaled.

Address assignment is done by CES Base Addresses. If the command is executed in the interface mode it is rejected, because CES cannot verify if CES Base Addresses are assigned.

interface

Set the cluster in CES interface mode and supports IPv6 addresses.

Address assignment is done by NIC configuration. The command is rejected if problems occur in the CES address assignments when this command is executed.

--clear

Remove the CES NIC configuration and set the cluster to legacy mode. This command requests user validation.

--force

Overwrites any check or request for user validation.

list

Shows the CES mode. The CES mode is either legacy or interface.

--by-node

Displays the CES mode for the specified node or nodes.

--by-group

Displays the CES mode for the specified group or groups.

ALL

Display all the NICs with at least one group assigned to them.

ANY

Displays all the NICs.

NONE

Displays all the NICs with no group assigned to them.

add

Add specified NICs to the cluster using the specified options. The command is rejected if problems occur in the CES address assignments when this command is executed.

--nic

Specify the NICs as a comma-separated list.

--ces-group

Sets all the NICs specified in this command into the specified group or groups. Groups are specified by a comma-separated list.

--ces-node

Limit the NIC assignment to the specified node or nodes. Nodes are specified by a comma-separated list.

--force

Disable checking for potential problems.

remove

Remove the specified NICs from the cluster using the specified options. The command is rejected if problems occur in the CES address assignments when this command is executed.

--nic

Specify the NICs as a comma-separated list.

--ces-node

Limit the NIC removal to the specified node or nodes. Nodes are specified by comma-separated list.

--force

Disable checking for potential problems.

check

Checks the NIC configuration against the node groups and CES addresses.

If the NIC configuration matches the node groups the configuration is valid. The command checks if any NIC is defined, and if there is any CES address that cannot be assigned to that NIC. You can check if a CES address cannot be assigned to a NIC by verifying if any of the NIC groups is assigned to a CES address. No checks are done for suspended nodes or for NICs that are currently down.

events

Shows one of the following CES events that occurred on a node or nodes:

active

Lists all events that are currently contributing to making the state of a component unhealthy. If no component is specified, active events for all components are listed. If neither the -N or -a parameters are specified, the active events for the local node are listed. If there are multiple events shown by the command they will be listed in the order we recommend they be fixed, with the most important event to fix at the top.

list

Lists the events that occurred on a node or nodes, whether or not they are currently contributing to the state of a component. If no component is specified, events for all components are listed. If --time is specified, only events from the previous chosen interval are listed, otherwise all events are listed. If --severity is specified, only events of the chosen severity are listed, otherwise all events are listed. If neither the -N or -a parameters are specified, the events for the local node are listed.

Events older than 180 days are removed from the list. A maximum of 10,000 events are saved in the list.

Exit status

0

Successful completion.

nonzero

A failure has occurred.

Security

You must have root authority to run the mmces command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.

See also

Location