Firewall recommendations for internal communication among nodes

The IBM Storage Scale system uses the following ports for internal communication among various IBM Storage Scale nodes.

Important: The ports that you plan to use for IBM Storage Scale internal communication might be blocked by a firewall or for some other reason on some nodes in a cluster. If so, then IBM Storage Scale communication errors will occur and some operations might fail. Therefore it is important to verify that the IBM Storage Scale internal communication ports on each node are accessible from every node in the cluster, including the node itself. Also, if you plan for nodes in one cluster to mount file systems in another cluster, then it is important to verify that all the IBM Storage Scale ports for internal communication in either cluster are accessible by all the nodes in the other cluster. If not, an attempt by a node in one cluster to mount a file system in another cluster might fail, or nodes in the remote cluster might be expelled.
Table 1. Recommended port numbers that can be used for internal communication
Port Number Protocol Service Name Components that are involved in communication
1191 TCP GPFS Intra-cluster
22 TCP Remote shell command, such as SSH.

Commands
22 TCP Remote file copy command, such as SCP. Commands
––- ICMP ICMP ECHO (ping). Intra-cluster
User-selected range TCP GPFS ephemeral port range Intra-cluster
  • The SSH and SCP port 22 is used for command execution and general node-to-node configuration as well as administrative access.
  • The GPFS Start of changeand CCREnd of change daemons (mmfsd and mmsdrserv), by default, listen on port 1191. This port is essential for basic cluster operation. The port can be changed manually by setting the Start of changetscTcpPortEnd of change configuration variable with the Start of changemmchconfig tscTcpPortEnd of change=PortNumber command.
  • The ephemeral port range of the underlying operating system is used when IBM Storage Scale creates additional sockets to exchange data among nodes. This occurs while executing certain commands and this process is dynamic based on the point in time needs of the command as well as other concurrent cluster activities. You can define an ephemeral port range manually by setting the tscCmdPortRange configuration variable with the mmchconfig tscCmdPortRange=LowNumber-HighNumber command.

If the installation toolkit is used, the ephemeral port range is automatically set to 60000-61000. Firewall ports must be opened according to the defined ephemeral port range. If commands such as mmlsmgr and mmcrfs hang, it indicates that the ephemeral port range is improperly configured.

For related information, see the topic IBM Storage Scale port usage.

The following are the recommendations for securing internal communications among IBM Storage Scale nodes:
  • Allow connection only to the GPFS cluster node IPs (internal IPs and protocol node IPs) on port 1191. Block all other external connections on this port. Use the mmlscluster --ces command to get the list of protocol node IP and use the mmlscluster command to get the list of IPs of internal nodes.
  • Allow all external communications request that are coming from the admin or management network and IBM Storage Scale internal IPs on port 22.
  • Certain commands such as mmadddisk, mmchmgr, and so on require an extra socket to be created for the duration of the command. The port numbers that are assigned to these temporary sockets are controlled with the tscCmdPortRange configuration parameter. If an explicit range is not specified, the port number is dynamically assigned by the operating system from the range of ephemeral port numbers. It is highly recommended to set the port range. For more information on how to set the port range, see IBM Storage Scale port usage.