Primary gateway and afmHashVersion

Each cache fileset in a cluster is served by one of the nodes that is designated as a gateway in the cluster. The gateway node that is mapped to a fileset is called the primary gateway of the fileset. The primary gateway acts as the owner of the fileset and communicates with the home cluster.

All other nodes in the cluster, including other gateways, become the application nodes of the fileset. Therefore, any node in the cache cluster can function as a gateway node and an application node for different filesets based on configuration of the node.

Application nodes communicate with the primary gateway for a fileset via internal network requests. The gateway function is highly available and can be scaled out. When a gateway node fails, all cache filesets that are owned by this gateway node are moved to another gateway node. This gateway node runs AFM recovery and takes over as the primary gateway of the filesets. When the old primary gateway returns after node failure, the original primary gateway resumes managing its fileset, and AFM transfers the queues for all the filesets from the current primary gateway to the old primary gateway.

The primary gateway can be configured to take help from other gateway nodes for parallel data movement of large files to and from home, if the home is an IBM Spectrum Scale cluster for increasing performance during data transfer. It is not recommended to assign the manager role to the gateway node because the assigned role can hamper the gateway node performance. The manager role can be assigned to other non-gateway nodes.

The gateway node designation is supported only on the Linux® operating system. The Windows or AIX® nodes cannot be designated as gateway.

The recommended setup configuration of a primary gateway node is as follows:

• The memory and CPU requirement for a gateway node depends on the number of assigned AFM or AFM-DR filesets and files inside the filesets. It is recommended to have minimum 128 GB of memory for the gateway node.
• Up to 20 AFM or AFM-DR filesets can be assigned to a gateway node.
• A gateway node is a dedicated node in the cluster without any other designations such as NSD server, CES protocol nodes, quorum, and manager.
• The recommended limit of the total number of inodes in all AFM, AFM-DR mode filesets that are assigned to the same gateway node is approximately 400 million.
• The following table contains the cluster level parameters:

  Parameter	Value
  Pagepool	8G
  afmHardMemThreshold	40G
  afmNumFlushThreads	8
  afmDIO	2
  maxFilestoCache	10000

  To set these parameters, use the mmchconfig command. For example,

  # mmchconfig pagepool=8G 

  For more information about configuration parameters, see Parameters for performance tuning and optimization.

  This configuration recommendation is based on observations in a controlled test environment by running moderate or reasonable workload. The parameters values might vary based on the setup or workload.

Note: AFM gateway node is licensed as a server node.

See Parallel data transfers to set up gateway nodes that communicate to multiple NFS servers at home.

afmHashVersion

In a cluster with multiple gateway nodes and AFM cache filesets, AFM uses a hashing algorithm to elect the primary gateway for each of the filesets. This algorithm can be selected by changing the cluster level config tunable afmHashVersion. Issue the following command to change the hashing algorithm.

# mmchconfig afmHashVersion=<version> 

Note: You need to shut down the cluster and then issue this command because you cannot set afmHashVersion on an active cluster.

AFM Hash Version 2

This version is the default version. The hashing algorithm is set by default on an IBM Spectrum Scale 4.1 cluster. This version is based on the static fileset ID mapping. On an upgraded cluster, the old hashing algorithm is effective by default. You can change the hashing version by changing the value of the afmHashVersion tunable.

AFM Hash Version 4

This version is an improved hash version '2'. If you have many filesets and if these filesets are not evenly distributed across gateway nodes, you can set the afmHashVersion parameter to 4 for the improvement. This hashing version is dynamic, which ensures balanced mapping of AFM filesets and gateway nodes.

Note: Do not set the afmHashVersion parameter to 4, if AFM has heavy data transfer operations from the gateway node to filesets. You are not recommended to set this version because you might face some deadlock issues in some cases. You can choose version '5' over version '4'.

AFM Hash Version 5

For better load balancing of AFM or AFM-DR filesets across gateway nodes, you can set the afmHashVersion parameter to '5'. This version is recommended for the load balancing of the filesets. AFM hash version '5' does not cause any known performance degradation in comparison with earlier versions. This option must set only at the cache or primary cluster and at the client cluster if it has remote mounts, that is, the AFM cache cluster. However, do not set this option at the home or secondary cluster because the home or primary cluster does not have AFM or AFM-DR filesets.

AFM hash version 5 supports manual assignment of the selected gateway node to an AFM or AFM-DR fileset by using the mmafmctl stop command and the mmafmctl start command. You can change the default assigned gateway node of an AFM or AFM-DR fileset by using the mmchfileset or mmcrfileset command. When the specified gateway node is down or not available, then AFM internally assigns a gateway node from the available gateway node list to the fileset. For filesets that do not have the afmGateway parameter set, the gateway node is assigned by using the hash version 2.

To assign the afmGateway parameter to a fileset, see the mmchfileset command or mmcrfileset command.

You must stop replication on an AFM or AFM-DR fileset before you set the afmGateway parameter and start replication after the parameter is set.

Do the following steps to assign a gateway node by using the mmchfileset command:

1. Stop replication on an AFM or AFM-DR fileset by issuing the following command:

   # mmafmctl fs stop -j AFM/AFM-DR Fileset

2. Set the afmGateway parameter by issuing the following command:

   # mmchfileset fs AFM/AFM-DR Fileset -p afmGateway=NewGatewayNode

3. Start replication on the AFM or AFM-DR fileset by issuing the following command:

   # mmafmctl fs start -j AFM/AFM-DR Fileset

To assign a gateway node by using the mmcrfileset command, issue the following command:

# mmcrfileset fs fileset -p afmMode=mode,afmTarget=Target,afmGateway=Gateway --inode-space new

For more information about start and stop replication on a fileset, see Stop and start replication on a fileset.

Note:

• To set the afmHashVersion parameter, IBM Spectrum Scale cluster must be shut down and start after this cluster level tunable is set for a new hash algorithm to take effect.
• To change the afmHashVersion parameter to ‘5’, all the nodes in the AFM cache and the client cluster must be upgraded to the minimum 5.0.2 level.