Multiple IBM Storage Scale File System support

This feature is available from HDFS Transparency version 2.7.3-1.

NameNode federation feature in Hadoop 3.x enables adding multiple native HDFS namespaces under single Hadoop cluster. This feature was added to improve the HDFS NameNode horizontal scaling. If you want to use native HDFS namespace and IBM Storage® Scale namespaces under the same Hadoop cluster, you can either use Hadoop Storage Tiering with IBM Storage Scale or ViewFS support. You can have multiple IBM Storage Scale file systems under single Hadoop cluster with ViewFS support. However, federation is not officially supported by HortonWorks HDP nor is it certificated with Hive in the Hadoop community. The multiple IBM Storage Scale file system support is designed to help resolve the federation issue.

If you want to enable this feature, you could configure the following properties in the /usr/lpp/mmfs/hadoop/etc/hadoop/gpfs-site.xml (for HDFS Transparency 2.7.3-x) or /var/mmfs/hadoop/etc/hadoop/gpfs-site.xml (for HDFS Transparency 3.0.x).

Configure the HDFS Transparency on the primary file system as the 1st file system defined in the gpfs.mnt.dir field.

Once the primary file system configuration is working properly, enable the multiple IBM Storage Scale file system support.

Note:

Currently multiple IBM Storage Scale file system only supports 2 file systems.
Do not put the NameNode and DataNode onto the same node when you are using multiple file system configuration. The reason for this being when either the NameNode or DataNode is stopped, the second file system gets unmounted and that leaves the other (still running) daemon non-functional.

Example of the gpfs-site.xml configuration for multiple file systems:

  <property>
    <name>gpfs.mnt.dir</name>
    <value>/path/fpo,/path/ess1</value>
  </property>
  <property>
    <name>gpfs.storage.type</name>
    <value>local,shared</value>
  </property>
  <property>
    <name>gpfs.replica.enforced</name>
    <value>dfs,gpfs</value>
  </property>

The gpfs.mnt.dir is a comma delimited string used to define the mount directory for each file system. In the above configuration, we have two file systems with mount point /path/fpo and /path/ess1. The first file system, /path/fpo will be considered as the primary file system.

The gpfs.storage.type is a comma delimited string used to define the storage type for each file system defined by gpfs.mnt.dir. Currently, only support 2 file systems mount access as local,shared or as shared,shared. The local means the file system with the same index in gpfs.mnt.dir is the IBM Storage Scale FPO file system with locality. The shared means the file system with the same index in gpfs.mnt.dir is the SAN-based file system or IBM Storage Scale System. You need to configure the gpfs.storage.type values correctly; otherwise, performance will be impacted. To check if the file system is local or shared, run mmlspool <fs-name> all -L to see whether the allowWriteaAffinity of the file system datapool is yes or no. If the value is yes, configure local for this file system. If the value is no, configure shared for this file system.

The gpfs.replica.enforced is a comma delimited string used to define the replica enforcement policy for all file systems defined by gpfs.mnt.dir.

Sync the above changes to all the HDFS Transparency nodes and restart HDFS Transparency. HDFS Transparency will mount all the non-primary file systems with bind mode into the primary file system.

In the above example, the primary file system is /path/fpo. The /path/fpo/<gpfs.data.dir> is the root directory for HDFS Transparency. The secondary file system /path/ess1 will be mapped as /path/fpo/<gpfs.data.dir>/ess1 directory virtually. Therefore, if you have /path/fpo/<gpfs.data.dir>/file1, /path/fpo/<gpfs.data.dir>/file2, /path/fpo/<gpfs.data.dir>/dir1, after mapping, the hadoop dfs -ls / will see /file1, /file2, /dir1 and /ess1. Use the hadoop dfs -ls /ess1 to list all files/directories under /path/ess1.

Note:

The gpfs.data.dir is a single directory and it is always configured for the primary file system.
In the example, if the directory /path/fpo/<gpfs.data.dir>/ess1 exists and is not empty, on starting HDFS Transparency, an exception will be reported about the /path/fpo/<gpfs.data.dir>/ess1 directory is not empty and will fail to start. To resolve this issue, rename the directory /path/fpo/<gpfs.data.dir>/ess1 or remove all the files under the /path/fpo/<gpfs.data.dir>/ess1/ directory so that the directory does not contain any contents.
When HDFS Transparency is stopped, the second file system that is not the primary file system (1st file system) gets unmounted. The unmount will remove the GPFS file system but not the local directory name used for the mount. Ensure that this local directory used for the mounting of the 2nd file system does not contain any data when you start HDFS Transparency. Otherwise, HDFS Transparency will not be able to restart.