Configuring HDFS federation

Perform this task to configure HDFS federation for HDFS 2.x.

About this task

To enable this feature, you must first configure and format HSFS federation, and then configure IBM® Spectrum Symphony. Here are the configuration requirements for IBM Spectrum Symphony:

  1. The hosts in the NameNode resource group and Secondary NameNode resource group must be consistent with the HDFS configuration.
  2. In the service profiles of NameNode and Secondary NameNode, the value of MinInstances and MaxInstances must be the same, and equal to the number of namespaces.

Procedure

  1. In the service profiles of NameNode (defined in the namenode.xml file) and Secondary NameNode (defined in the secondarynode.xml file), set NAMESPACE_BASE_NAME as an environment variable.
  2. Open the hdfs-site.xml file. Add the namespace names to the dfs.nameservices property.

    The namespace names must take the form NAMESPACE_BASE_NAME and ID, where ID represents consecutive positive integers.

    For example, if environment variable NAMESPACE_BASE_NAME is set to namespace, edit the dfs.nameservices property as follows:
    <property>        
    <name>dfs.nameservices</name>        
    <value>namespace1,namespace2</value>   
</property>
  3. In the service profiles of NameNode, Secondary NameNode, and DataNode, set environment variable FEDERATION_VIP_ENABLE to true.
  4. Add hosts that run the NameNode daemon to resource group NameNodeRG.
  5. Add hosts that run the Secondary NameNode daemon to resource group SecondaryNodeRG.

Example

NameNode service profile:
...
<ego:EnvironmentVariable name="DATANODE_SERVICE">DataNode</ego:EnvironmentVariable>    
<ego:EnvironmentVariable name="SECONDARYNODE_SERVICE">SecondaryNode</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="NAMESPACE_BASE_NAME">namespace</ego:EnvironmentVariable>    
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
...
Secondary Name Node service profile:
...
<ego:EnvironmentVariable name="DATANODE_SERVICE">DataNode</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="SECONDARYNODE_SERVICE">SecondaryNode</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="NAMESPACE_BASE_NAME">namespace</ego:EnvironmentVariable>    
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
Data Node service profile:
...
<ego:EnvironmentVariable name="NAMENODE_SERVICE">NameNode</ego:EnvironmentVariable>
<ego:EnvironmentVariable name="FEDERATION_VIP_ENABLE">true</ego:EnvironmentVariable>
...