mmces command

Manages CES configuration.

Synopsis

mmces address add [--ces-node Node] [--attribute Attribue] [--ces-group Group] {--ces-ip {IP[,IP...]}

or

mmces address remove --ces-ip {IP[,IP...]}

or

mmces address move --ces-ip {IP[,IP...]} --ces-node Node

or

mmces address move --rebalance

or

mmces address change
                     {--ces-ip IP  | --remove-attribute}
                     --attribute Attribute[,Attribute...]
mmces address change
                     [--ces-ip IP[,IP...]]
                     [--attribute Attribute[,Attribute...]]
                     [--ces-group Group]
                     [--remove-attribute]
                     [--remove-group]

or

mmces address list [-N {Node[,Node...]| NodeFile | NodeClass}]
mmces address list [--ces-ip IP[,IP...]] [--ces-group Group[,Group...]]
                   [--attribute Attribute[,Attribute...]] [--by-node]

or

mmces address policy [none | balanced-load | node-affinity | even-coverage]

or

mmces node {suspend | resume} [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

or

mmces node list [--ces-group Group[,Group...]] [--verbose]

or

mmces service {enable | disable} {NFS | SMB | OBJ | BLOCK}

or

mmces service {start | stop} {NFS | SMB | OBJ | BLOCK}
              [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

or

mmces service list [-N {Node[,Node...] | NodeFile | NodeClass} | -a] [--verbose]

or

mmces log level [new-level]

or

mmces events active [NFS | OBJ | SMB | AUTH | NETWORK]
             [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

or

mmces events list [NFS | OBJ | SMB | AUTH | NETWORK]
             [--time {hour | day | week | month}]
             [--severity {INFO | WARNING | ERROR | SEVERE}]
             [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

or

mmces state show [NFS | OBJ | SMB | AUTH | NETWORK | CES]
            [-N {Node[,Node...] | NodeFile | NodeClass} | -a]

or

mmces state cluster [NFS | OBJ | SMB | AUTH | AUTH_OBJ  | NETWORK | CES]

Availability

Available with IBM Spectrum Scale™ Standard Edition or higher.

Description

Use the mmces command to manage protocol addresses, services, node state, logging level and load balancing. The protocol functions provided in this command, or any similar command, are generally referred to as CES (Cluster Export Services). For example, protocol node and CES node are functionally equivalent terms.

CES currently supports the NFS, SMB, BLOCK, and Object services. Each service can be enabled or disabled with the mmces service command. Enabling a service is a CES cluster-wide operation. In addition, enabled services can be started and stopped on individual nodes.

Clients access the CES services using one or more IP addresses in the CES address pool. Addresses can be added to and removed from the pool using the mmces address add and mmces address remove commands. Existing addresses can be reassigned to another node with the mmces address move command.

Addresses can have one or more attributes associated with them. An address attribute is a tag that the services can identify a specific address as having a special meaning, which is defined by the service protocol. Addresses can have multiple attributes, but an attribute can only be associated with a single address.

Addresses can have a policy associated with them, and that policy determines how addresses are automatically distributed. The allowed policies are none, balanced-load, node-affinity, and even-coverage. A policy of none means addresses are not distributed automatically.

A CES node can be placed in a suspended state. When a node is in suspended state, all of the CES addresses for that node are reassigned to other nodes, and the node will not accept new address assignments. Any services that are started when the node is suspended continue to run. The suspended state is persistent, which means nodes remain suspended following a reboot. Use the mmces node command to suspend and resume a node.

Parameters

address
Manages CES addresses with one of the following actions:
add
Adds the addresses specified by the --ces-ip parameter to the CES address pool and assigns them to a node. The node to which an address is assigned will configure its network interfaces to accept network communication destined for the address. CES addresses must be different from IP addresses used for GPFS™ or CNFS communication.

If --ces-node is specified with add, all addresses specified with the --ces-ip parameter will be assigned to this node. If --ces-node is not specified, the addresses will be distributed among the CES nodes.

If an attribute is specified with --attribute, there can only be one address specified with the --ces-ip parameter.

If --ces-group is specified with add, all new addresses will be associated with the specified group. The result can be viewed with the "mmces address list" command.

remove
Removes the addresses specified by the --ces-ip parameter from the CES address pool. The node to which the address is assigned will reconfigure its network interfaces to no longer accept communication destined for the address. The node specified in the command will configure its network interfaces to accept communication for this address.
move
Moves addresses.

If the --ces-ip parameter is specified, the addresses specified by IP are moved from one CES node to another. The addresses are reassigned to the node specified by the --ces-node parameter.

If the --rebalance parameter is specified, the addresses are distributed immediately based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.

Use this command with caution because IP movement will trigger CES protocol recovery.

change
Changes or removes address attributes.
If the --ces-ip parameter is specified:
  • the attributes specified by the --attribute parameter are associated with the address specified by the --ces-ip parameter. If an attribute was previously associated with another address, that association is ended.
  • the attributes specified by the --attribute parameter, with the --remove-attribute parameter, have the specified attributes removed from the addresses specified by the --ces-ip parameter.
  • groups specified by the --ces-group parameter are associated with the address specified by the --ces-ip parameter.
  • groups specified by the --ces-group parameter, with the --remove-group parameter, have the specified groups removed from the addresses specified by the --ces-ip parameter and is no longer used.
If the --ces-ip parameter is not specified:
  • specifying --remove-attribute, with the attributes specified by the attributes specified by the --attribute parameter, removes the attributes from their current associations and they are nolonger used.
  • specifying --remove-group, with the groups s specified by the --group parameter, removes the groups from their current associations and they are nolonger used.
list
Lists the CES addresses along with group, attribute and node assignments.
Options:
  • --ces-ip Only list the addresses provided.
  • --ces-group Only list addresses whose group assignment matches one of the groups provided.
  • --attribute Only list addresses whose attributes mach one of the attributes provided.
  • --ces-node Only list addresses assigned to one of the nodes provided.
  • --by-node List addresses by node, using the output format from IBM Spectrum Scale V4.1.1 and later.
policy
Sets the CES address distribution policy.
node
Manages CES node state with one of the following actions:
suspend
Suspends the specified nodes. If neither the -N or -a parameters are specified, only the local node is suspended.

When a node is suspended, all addresses assigned to the node are reassigned to other nodes and the node will not accept any subsequent address assignments. Suspending a node will trigger CES protocol recovery if the node has CES addresses assigned.

resume
Resumes the specified suspended nodes. If neither the -N or -a parameters are specified, only the local node is resumed.
When a suspended node is resumed (no longer suspended), the node will accept subsequent address assignments.
list
Lists the specified nodes along with their current node state. If the -N parameter is not specified, all nodes are listed.
verbose
Lists the addresses assigned to the nodes.
--ces-group
Lists the nodes belonging to the specified groups.
service
Manages protocol services with one of the following actions:
enable
Enables and starts the specified service on all CES nodes.
disable
Disables and stops the specified service on all CES nodes.
Note: Disabling a service will discard any configuration data from the CES cluster and needs to be used with caution. If applicable, backup any relevant configuration data. Subsequent service enablement will start with a clean configuration.
start
Starts the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is started on the local node.
stop
Stops the specified service on the nodes specified. If neither the -N or -a parameters are specified, the service is stopped on the local node.
Note: If a service is stopped on a node that has CES addresses assigned, clients will not be able to access the service using any of the addresses assigned to that node. Access to the data from clients is not possible any more for services that are stopped.
list
Lists the state of the enabled services.
log level
Sets and checks the CES log level. The CES log level determines how much information related to the CES nodes is entered into the GPFS log file. Values can be from 0 (less logging) to 3 (more logging).
events
Shows one of the following CES events that occurred on a node or nodes:
active
Lists all events that are currently contributing to making the state of a component unhealthy. If no component is specified, active events for all components are listed. If neither the -N or -a parameters are specified, the active events for the local node are listed. If there are multiple events shown by the command they will be listed in the order we recommend they be fixed, with the most important event to fix at the top.
list
Lists the events that occurred on a node or nodes, whether or not they are currently contributing to the state of a component. If no component is specified, events for all components are listed. If --time is specified, only events from the previous chosen interval are listed, otherwise all events are listed. If --severity is specified, only events of the chosen severity are listed, otherwise all events are listed. If neither the -N or -a parameters are specified, the events for the local node are listed.
Events older than 180 days are removed from the list. A maximum of 10,000 events are saved in the list.
state
Shows the state of one or more nodes in the cluster.
show
Shows the state of the specified service on the nodes specified. If no service is specified, all services will be displayed. If neither the -N or -a parameters are specified, the state of the local node is shown.
cluster
Shows the combined state for the services across the whole CES cluster. If no service is specified an aggregated state will be displayed for each service, where healthy means the service is healthy on all nodes, degraded means the service is not healthy on one or all nodes, and failed means that the service is not available on any node. If a service is specified the state of that service will be listed for each node, along with the name of any event that is contributing to an unhealthy state.
--ces-node
Indicates that the command applies only to the specified CES node name.
--attribute
Specifies either a single attribute or a comma-separated list of attributes as indicated in the command syntax.
--ces-ip
Specifies either a single or comma-separated list of DNS qualified host names or IP addresses as indicated in the command syntax.
--rebalance
Distributes addresses immediately based on the currently configured distribution policy. If the policy is currently undefined or none, the even-coverage policy is applied.
none
Specifies that addresses are not distributed automatically.
balanced-load
Distributes addresses dynamically in order to approach an optimized load distribution throughout the cluster. The network and CPU load on all the nodes is monitored and addresses are moved based on given policies.

Addresses that were recently moved or addresses with attributes are not moved.

node-affinity
Attempts to keep addresses associated with the node to which they were assigned. Address node associations are created with the --ces-node parameter of the mmces address add command or the mmces address move command. Automatic movements of addresses do not change the association. Addresses that were enabled without a node specification do not have a node association. Addresses that are associated with a node but assigned to a different node are moved back to the associated node.

Addresses that were recently moved or addresses with attributes are not moved.

even-coverage
Attempts to evenly distribute all of the addresses among the available nodes.

Addresses that were recently moved or addresses with attributes are not moved.

--remove-attribute
Indicates that the specified attributes should be removed.
-N {Node[,Node...] | NodeFile | NodeClass}
Indicates that the command applies only to the specified node names.

For general information on how to specify node names, see Specifying nodes as input to GPFS commands in the IBM Spectrum Scale: Administration and Programming Reference.

-a
Specifies that the command applies to all CES nodes.
NFS
Specifies that the command applies to the NFS service.
OBJ
Specifies that the command applies to the Object service.
SMB
Specifies that the command applies to the SMB service.
AUTH
Specifies that the command applies to the AUTH service.
NETWORK
Specifies that the command applies to the NETWORK service.
CES
Specifies that the command applies to the CES service.
--verbose
Specifies that the output is verbose.
new-level
Sets the log level to a new value. If the new-level parameter is not specified, the current log level is displayed.
--time
Lists the previous events from one of the following intervals:
hour
Lists the events from the past hour.
day
Lists the events from the past day.
week
Lists the events from the past week.
month
Lists the events from the past month.
The events are listed whether or not they are currently contributing to the state of a component.
--severity
Specifies that only events for one of the following severities are listed:
INFO
Lists only informational events.
WARNING
Lists only warning events.
ERROR
Lists only error events.
SEVERE
Lists only severe events.

Exit status

0
Successful completion.
nonzero
A failure has occurred.

Security

You must have root authority to run the mmces command.

The node on which the command is issued must be able to execute remote shell commands on any other node in the cluster without the use of a password and without producing any extraneous messages. For more information, see Requirements for administering a file system in IBM Spectrum Scale: Administration and Programming Reference.

Examples

  1. To add an address to a specified node, issue this command:
    mmces address add --ces-node node1 --ces-ip 10.1.2.3
    When this command is successful, the system does not display output.
  2. To add several addresses to a specified node, issue this command:
    mmces address add --ces-node node1 --ces-ip 10.1.2.3,10.1.2.4
    When this command is successful, the system does not display output.
  3. To add an address with the attribute xyz_server to a specified node, issue this command:
    mmces address add --ces-node node1 --ces-ip 10.1.2.3 --attribute xyz_server
    When this command is successful, the system does not display output.
  4. To add addresses which are distributed among the CES nodes, issue this command:
    mmces address add --ces-ip 10.1.2.3,10.1.2.4,10.1.2.5,10.1.2.6
    When this command is successful, the system does not display output.
  5. To remove several addresses, issue this command:
    mmces address remove --ces-ip 10.1.2.3,10.1.2.4
    When this command is successful, the system does not display output.
  6. To associate the attribute xyz_server to the address 10.1.2.3, issue this command:
    mmces address change --ces-ip 10.1.2.3 --attribute xyz_server
    When this command is successful, the system does not display output.
  7. To remove the attribute xyz_server, issue this command:
    mmces address change --remove-attribute --attribute xyz_server
    When this command is successful, the system does not display output.
  8. To move an address to another node, issue this command:
    mmces address move --ces-ip 10.0.100.231 --ces-node node2
    When this command is successful, the system does not display output.
  9. To suspend a group of nodes, issue this command:
    mmces node suspend -N node1,node2,node3
    The system displays output similar to this:
    Node now in suspended state.
  10. To resume the current node, issue this command:
    mmces node resume
    The system displays output similar to this:
    Node no longer in suspended state.
  11. To enable the Object service in the CES cluster, issue this command:
    mmces service enable obj
    When this command is successful, the system does not display output.
  12. To disable the NFS service in the CES cluster, issue this command:
    mmces service disable nfs
    When this command is successful, the system does not display output.
  13. To stop the SMB service on a few nodes, issue this command:
    mmces service stop smb -N node1,node2,node3
    When this command is successful, the system does not display output.
  14. To start the SMB service on all CES nodes, issue this command:
    mmces service start smb -a
    When this command is successful, the system does not display output.
  15. To show which services are enabled and which are running all CES nodes, issue this command:
    mmces service list -a
    The system displays output similar to this:
    Enabled services: NFS OBJ
node1: NFS is running, OBJ is running
node2: NFS is running, OBJ is running
node3: NFS is running, OBJ is running
  16. To display the current CES log level, issue this command:
    mmces log level
    The system displays output similar to this:
    CES log level is currently set to 1
  17. To set the CES log level to 2, issue this command:
    mmces log level 2
    The system displays output similar to this:
    mmchconfig: Command successfully completed
mmchconfig: Propagating the cluster configuration data to all
  affected nodes. This is an asynchronous process.
  18. To display the state of all CES components on the local node, issue this command:
    mmces state show
    The system displays output similar to this:
    NODE    AUTH      AUTH_OBJ     NETWORK    NFS       OBJ       SMB        CES
node1   HEALTHY   DISABLED     HEALTHY    HEALTHY   HEALTHY   DISABLED   HEALTHY
  19. To display the state of the NFS component on all nodes, issue this command:
    mmces state cluster NFS
    The system displays output similar to this:
    NODE   COMPONENT  STATE    EVENTS
node1  NFS        HEALTHY
node2  NFS        FAILED   nfsd_down
node3  NFS        HEALTHY
  20. To display a list of active events of all CES components on the local node, issue this command:
    mmces events active
    The system displays output similar to this:
    Node    Component   Event Name   Severity     Details
node1   NFS         nfsd_down    ERROR        NFSD process not running
  21. To display a list of all NFS events from the last hour on the local node, issue this command:
    mmces events list
    The system displays output similar to this:
    Node  Timestamp                           Event Name              Severity Details
node1 2015-05-13 10:57:52.124369+00:00UTC nfsd_down               ERROR    NFSD process not running
node1 2015-05-13 10:58:06.809071+00:00UTC cesnodestatechange_info INFO     A CES node state changed
node1 2015-05-13 10:58:07.137343+00:00U   ganeshagrace_info       INFO     Ganesha NFS is set to grace

See also

Location

/usr/lpp/mmfs/bin
Parent topic: GPFS commands