mmces command

Manage CES (Cluster Export Services) configuration.

Synopsis

mmces address add [--ces-node Node] [--attribute Attribue]
                 [--ces-group Group] {--ces-ip {IP[,IP...]}

mmces address remove --ces-ip {IP[,IP...]}

mmces address move --ces-ip {IP[,IP...]} --ces-node Node

mmces address move --rebalance

mmces address change
                     {--ces-ip IP  | --remove-attribute}
                     --attribute Attribute[,Attribute...]

mmces address change
                     [--ces-ip IP[,IP...]]
                     [--attribute Attribute[,Attribute...]]
                     [--ces-group Group]
                     [--remove-attribute]
                     [--remove-group]

mmces address list [-N {Node[,Node...]| NodeFile | NodeClass} | -a] [-Y]

mmces address list [--ces-ip IP[,IP...]] [--ces-group Group[,Group...]]
                  [--attribute Attribute[,Attribute...]] 
                  [--by-node | [--extended-list | --full-list] -Y]

mmces address policy [none | balanced-load | node-affinity | even-coverage]

mmces node {suspend [--stop ] | resume [--start ]}
           [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

mmces node list [-Y] [--ces-group Group[,Group...]] [--verbose]

mmces service enable  {NFS | SMB | OBJ | BLOCK}

mmces service disable {NFS | SMB | OBJ | BLOCK} [--force]

mmces service {start | stop} {NFS | SMB | OBJ | BLOCK}
              [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

mmces service list [-N {Node[,Node...] | NodeFile | NodeClass} | -a] [-Y] [--verbose]

mmces log level [new-level]

mmces events active [NFS | OBJ | SMB | BLOCK | AUTH | AUTH_OBJ |NETWORK]
             [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

mmces events list [NFS | OBJ | SMB | BLOCK | AUTH | AUTH_OBJ |NETWORK]
             [-Y] [--time {hour | day | week | month}]
             [--severity {INFO | WARNING | ERROR | SEVERE}]
             [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

mmces state show [NFS | OBJ | SMB | BLOCK | AUTH | AUTH_OBJ |NETWORK | CES]
            [-N {Node[,Node...] | NodeFile | NodeClass} | -a] [-Y]

mmces state cluster [NFS | OBJ | SMB | BLOCK | AUTH | AUTH_OBJ | NETWORK | CES] [-Y]

Availability

Available on all IBM Spectrum Scale editions.

Description

Use the mmces command to manage protocol addresses, services, node state, logging level and load balancing. The protocol functions provided in this command, or any similar command, are generally referred to as CES (Cluster Export Services). For example, protocol node and CES node are functionally equivalent terms. Before using the mmces command, CES has to be installed and configured in the system. For more information on how to install CES, see Installing Cluster Export Services as part of installing IBM Spectrum Scale on Linux systems.

CES currently supports the NFS, SMB, BLOCK, and Object services. Each service can be enabled or disabled with the mmces service command. Enabling a service is a CES cluster-wide operation. In addition, enabled services can be started and stopped on individual nodes.

Clients access the CES services using one or more IP addresses in the CES address pool. Addresses can be added to and removed from the pool using the mmces address add and mmces address remove commands. Existing addresses can be reassigned to another node with the mmces address move command.

Addresses can have one or more attributes associated with them. An address attribute is a tag that the services can identify a specific address as having a special meaning, which is defined by the service protocol. Addresses can have multiple attributes, but an attribute can only be associated with a single address. The supported attributes are object_singleton_node and object_database_node. These attributes are used to manage object protocol-related services. For more information, see the Object services and object protocol nodes table in Understanding and managing Object services.

Addresses can have a policy associated with them, and that policy determines how addresses are automatically distributed. The allowed policies are none, balanced-load, node-affinity, and even-coverage. A policy of none means addresses are not distributed automatically.

A CES node can be placed in a suspended state. When a node is in suspended state, all of the CES addresses for that node are reassigned to other nodes, or set to unassigned. The node will not accept new address assignments. Any services that are started when the node is suspended continue to run, except mmcesmonitord. The mmcesmonitord service is stopped, and the lock on the sharedRoot file system is released. The sharedRoot file system is still mounted, but can be unmounted. The suspended state is persistent, which means nodes remain suspended following a reboot. Start of change After a reboot no services are running on a suspended node. However, if a node is not suspended, the services that were enabled on the nodes are restarted after the reboot. End of change

Parameters

address

Manages CES addresses with one of the following actions:

add

Adds the addresses specified by the --ces-ip parameter to the CES address pool and assigns them to a node. The node to which an address is assigned will configure its network interfaces to accept network communication destined for the address. CES addresses must be different from IP addresses used for GPFS or CNFS communication.

If --ces-node is specified with add, all addresses specified with the --ces-ip parameter will be assigned to this node. If --ces-node is not specified, the addresses will be distributed among the CES nodes.

If an attribute is specified with --attribute, there can only be one address specified with the --ces-ip parameter.

If --ces-group is specified with add, all new addresses will be associated with the specified group. The result can be viewed with the mmces address list command.

Note: The provided addresses or host names must be resolvable by forward and reverse name resolution (DNS or /etc/hosts on all CES nodes). Otherwise you get the following error message:

Cannot resolve <ip address>; Name or service not known

You can also perform a manual check by running the following command: mmcmi host <ip address>.

Ensure that the netmask (PREFIX) setting in the ifcfg-<interface> files is correct.

remove

Removes the addresses specified by the --ces-ip parameter from the CES address pool. The node to which the address is assigned reconfigures its network interfaces to no longer accept communication for that address.

move

Moves addresses.

If the --ces-ip parameter is specified, the addresses specified by IP are moved from one CES node to another. The addresses are reassigned to the node specified by the --ces-node parameter.

If the --rebalance parameter is specified, the addresses are distributed within 60 seconds based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.

Note: The information about the address movement is printed after the rebalance is done. The address movement is also done in the background periodically.

Use this command with caution because IP movement will trigger CES protocol recovery.

change

Changes or removes address attributes.

If the --ces-ip parameter is specified:

The command associates the attributes that are specified by the --attribute parameter with the address that is specified by the --ces-ip parameter. If an attribute is already associated with another address, that association is ended.
If the --remove-attribute parameter is specified, the command removes the attributes that are specified by the --attribute parameter from the addresses that are specified by the --ces-ip parameter.
The command associates the groups that are specified by the --ces-group parameter with the address that is specified by the --ces-ip parameter
If the --remove-group parameter is specified, the command removes the groups that are specified by the --ces-group parameter from the addresses that are specified by the --ces-ip parameter.

If the --ces-ip parameter is not specified:

If the --remove-attribute parameter is specified, the command removes the attributes that are specified by the --attribute parameter from their current associations.
If the --remove-group parameter is specified, the command removes the groups that are specified by the --group parameter from their current associations.
Specifying --remove-group with the groups specified by the --group parameter removes the groups from their current associations.

list

Lists the CES addresses along with group, attribute and node assignments.

Options:

--ces-ip List only the addresses provided.
--ces-group List only addresses whose group assignment matches one of the groups provided.
--attribute List only addresses whose attributes match one of the attributes provided.
--by-node List addresses by node, using the output format from IBM Spectrum Scale V4.1.1 and later.
--extended-list Lists the preferred node of the given address in a new column if the address balancing mode option is set to node-affinity.
--full-list Lists the information about the preferred node and the node names where the given address could not be hosted in two new columns.

Note: The [-N {Node[,Node...]| NodeFile | NodeClass| -a] option is deprecated for IBM Spectrum Scale version 4.2.3 and might be removed in a later release.

policy

Sets the CES address distribution policy.

node

Manages CES node state with one of the following actions:

suspend

Suspends the specified nodes. If neither the -N or -a parameters are specified, only the local node is suspended. The -a stands for all CES nodes.

When a node is suspended, all addresses assigned to the node are reassigned to other nodes, or set to unassigned. The node will not accept any subsequent address assignments. Suspending a node triggers CES protocol recovery if the node has CES addresses assigned. This does not stop any services or the mmces monitor daemon.

If the --stop option is added to the suspend command, all the CES services including the mmces monitor daemon are stopped. This releases the lock on the sharedRoot file system, and it can be unmounted without further actions. The sharedRoot file system is still mounted, but can be unmounted.

Note: After a reboot no services are running on a suspended node. However, if a node is not suspended, the services that were enabled on the nodes are restarted after the reboot.

resume

Resumes the node so that CES IPs can be retrieved and starts the mmces monitor daemon if it is not running.

Using the --start option starts all the enabled protocols. The mmces monitor daemon is also started, if it is not already running. However, if at least one protocol was not successfully started, then all the protocols that were already running are also stopped.

list

Lists the specified nodes along with their current node state. If the -N parameter is not specified, all nodes are listed.

verbose

Lists the addresses assigned to the nodes.

--ces-group

Lists the nodes belonging to the specified groups.

service

Manages protocol services with one of the following actions:

enable

Enables and starts the specified service on all CES nodes.

disable

Disables and stops the specified service on all CES nodes.

Note: Disabling a service will discard any configuration data from the CES cluster and needs to be used with caution. If applicable, backup any relevant configuration data. Subsequent service enablement will start with a clean configuration.

--force: Allows the service disable action to go through without confirmation from the user.

start

Starts the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is started on the local node.

stop

Stops the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is stopped on the local node.

Note: If a service is stopped on a node that has CES addresses assigned, clients will not be able to access the service using any of the addresses assigned to that node. Access to the data from clients is not possible any more for services that are stopped. This state is not persistent, so after a reboot all the services become active again.

list

Lists the state of the enabled services.

log level

Sets and checks the CES log level. The CES log level determines how much information related to the CES nodes is entered into the GPFS log file. Values can be from 0 (less logging) to 3 (more logging).

events

Shows one of the following CES events that occurred on a node or nodes:

active: Lists all events that are currently contributing to making the state of a component unhealthy. If no component is specified, active events for all components are listed. If neither the -N or -a parameters are specified, the active events for the local node are listed. If there are multiple events shown by the command they will be listed in the order we recommend they be fixed, with the most important event to fix at the top.
list: Lists the events that occurred on a node or nodes, whether or not they are currently contributing to the state of a component. If no component is specified, events for all components are listed. If --time is specified, only events from the previous chosen interval are listed, otherwise all events are listed. If --severity is specified, only events of the chosen severity are listed, otherwise all events are listed. If neither the -N or -a parameters are specified, the events for the local node are listed.; Events older than 180 days are removed from the list. A maximum of 10,000 events are saved in the list.

state

Shows the state of one or more nodes in the cluster.

show: Shows the state of the specified service on the nodes specified. If no service is specified, all services will be displayed. If neither the -N or -a parameters are specified, the state of the local node is shown.

cluster: Shows the combined state for the services across the whole CES cluster. If no service is specified an aggregated state will be displayed for each service, where healthy means the service is healthy on all nodes, degraded means the service is not healthy on one or all nodes, and failed means that the service is not available on any node. If a service is specified the state of that service will be listed for each node, along with the name of any event that is contributing to an unhealthy state.

--ces-node

Indicates that the command applies only to the specified CES node name.

--attribute

Specifies either a single attribute or a comma-separated list of attributes as indicated in the command syntax.

--ces-ip

Specifies either a single or comma-separated list of DNS qualified host names or IP addresses as indicated in the command syntax.

--rebalance

Distributes addresses immediately based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.

none

Specifies that addresses are not distributed automatically.

balanced-load

Distributes addresses dynamically in order to approach an optimized load distribution throughout the cluster. The network and CPU load on all the nodes is monitored and addresses are moved based on given policies.

Addresses that were recently moved or addresses with attributes are not moved.

node-affinity

Attempts to keep addresses associated with the node to which they were assigned. Address node associations are created with the --ces-node parameter of the mmces address add command or the mmces address move command. Automatic movements of addresses do not change the association. Addresses that were enabled without a node specification do not have a node association. Addresses that are associated with a node but assigned to a different node are moved back to the associated node.

Addresses that were recently moved or addresses with attributes are not moved.

even-coverage

Attempts to evenly distribute all of the addresses among the available nodes.

Addresses that were recently moved or addresses with attributes are not moved.

--remove-attribute

Indicates that the specified attributes should be removed.

-N {Node[,Node...] | NodeFile | NodeClass}

Indicates that the command applies only to the specified node names.

For general information on how to specify node names, see Specifying nodes as input to GPFS commands.

-a

Specifies that the command applies to all CES nodes.

NFS

Specifies that the command applies to the NFS service.

OBJ

Specifies that the command applies to the Object service.

SMB

Specifies that the command applies to the SMB service.

BLOCK

Specifies that the command applies to the BLOCK service.

AUTH

Specifies that the command applies to the AUTH service.

NETWORK

Specifies that the command applies to the NETWORK service.

CES

Specifies that the command applies to the CES service.

--verbose

Specifies that the output is verbose.

new-level

Sets the log level to a new value. If the new-level parameter is not specified, the current log level is displayed.

-Y

Displays the command output in a parseable format with a colon (:) as a field delimiter. Each column is described by a header.

Note: Fields that have a colon (:) are encoded to prevent confusion. For the set of characters that might be encoded, see the command documentation of mmclidecode. Use the mmclidecode command to decode the field.

--time

Lists the previous events from one of the following intervals:

hour: Lists the events from the past hour.
day: Lists the events from the past day.
week: Lists the events from the past week.
month: Lists the events from the past month.

The events are listed whether or not they are currently contributing to the state of a component.

--severity

Specifies that only events for one of the following severities are listed:

INFO: Lists only informational events.
WARNING: Lists only warning events.
ERROR: Lists only error events.
SEVERE: Lists only severe events.

Exit status

0: Successful completion.
nonzero: A failure has occurred.

Security

You must have root authority to run the mmces command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a GPFS file system.