Cluster Export Services overview

Cluster Export Services (CES) provide highly available file and object services to a GPFS cluster by using Network File System (NFS), Simple Storage Service (S3), Hadoop Distributed File System (HDFS), or Server Message Block (SMB) protocols.

IBM Storage Scale includes Cluster Export Services (CES) infrastructure to support the integration of the NFS, S3, HDFS, and SMB servers.

  • The NFS server supports NFS v3 and the mandatory features in NFS v4.0 and NFS v4.1.
  • The SMB server supports SMB 2, SMB 2.1, and the mandatory features of SMB 3.0 and 3.11.
  • The S3 server supports S3 APIs to access the data stored in IBM Storage Scale file systems.
  • CES supports HDFS protocols. HDFS Transparency is supported from IBM Storage Scale version 3.1.1. The IBM Storage Scale Big Data Analytics Integration Toolkit for HDFS Transparency or Toolkit for HDFS must be installed. The Toolkit for HDFS is supported from version 1.0.0.0. See Support Matrix under CES HDFS in Big data and analytics support documentation.

The CES infrastructure is responsible for:

  • Managing the setup for high-availability clustering that is used by the protocols.
  • Monitoring the health of these protocols on the protocol nodes and raising events or alerts during failures.
  • Managing the addresses that are used for accessing these protocols by including failover and failback of these addresses because of protocol node failures.

High availability

With GPFS, you can configure a subset of nodes in the cluster to provide a highly available solution for exporting GPFS file systems by using the Network File System (NFS), Server Message Block (SMB), Simple Storage Service (S3), and Hadoop Distributed File System (HDFS) protocols. The participating nodes are designated as Cluster Export Services (CES) nodes or protocol nodes. The set of CES nodes is frequently referred to as the CES cluster.

A set of IP addresses, the CES address pool, is defined and distributed among the CES nodes. As nodes enter and leave the GPFS cluster, the addresses in the pool can be redistributed among the CES nodes to provide high availability. GPFS nodes that are not CES nodes that are leaving or entering the GPFS cluster do not redistribute the IP addresses. However, clients that use the SMB, NFS, and S3 protocols must not share the IP address for these protocols with the IP addresses that are used by the HDFS service to avoid impacting the clients of other protocols during an HDFS failover. However, applications that use the CES IP addresses are not impacted. SMB failover is not transparent.

Monitoring

CES monitors the state of the protocol services. Monitoring ensures that the CES addresses are assigned to the appropriate node and that the processes that implement the services in the CES cluster are running correctly. Upon failure detection, CES marks the node as temporarily unable to participate in the CES cluster and reassigns the IP addresses to another node.

Protocol support

CES supports the following export protocols: NFS, SMB, S3, and HDFS. Each protocol can be enabled or disabled in the cluster. If a protocol is enabled in the CES cluster, all CES nodes serve that protocol.

The following are examples of enabling and disabling protocol services by using the mmces command:
mmces service enable nfs
Enables the NFS protocol in the CES cluster.
mmces service disable s3
Disables the S3 protocol in the CES cluster.

Commands

To set or configure CES options, use the following commands:
mmces
Manages the CES address pool and other CES cluster configuration options.
mmhdfs
Manages the HDFS configuration operations.
mmnfs
Manages NFS exports and sets the NFS configuration attributes.
mms3
Manages Simple Storage Service (S3) configuration, accounts, and buckets.
mmsmb
Manages SMB exports and sets the SMB configuration attributes.
mmuserauth
Configures the authentication methods that are used by the protocols.

For more information, see mmces command, mmhdfs command, mmnfs command, mms3 command, mmobj command, mmsmb command, and mmuserauth command.

Restrictions

  • For an up-to-date list of supported operating systems, specific distributions, and other dependencies for CES, refer to the IBM Storage Scale FAQ in IBM® Documentation.
  • A CES cluster can contain a maximum of 32 CES nodes. All the nodes in a CES cluster must be running on the same operating system and they must be of the same CPU architecture. If the SMB protocol is enabled, then the CES cluster is limited to a total of 16 CES nodes.
  • Each CES node must have network adapters capable of supporting all IP addresses in the CES address pool. The primary address of these adapters must not be used as a CES address.
  • The S3 protocol is tested with 10 CES nodes in the cluster.
  • Do not configure CES IP addresses in a subnet that is included in the 'subnets=' parameter. Because the CES IP addresses move across nodes, if this restriction is not heeded, the mmfsd server could fail to connect to another node.