Configuring ViewFs on HDFS cluster with HA

Edit online

The local native HDFS HA cluster is setup through Ambari. The local HDFS and remote HDFS both have HA setup. Either cluster can have Scale service added. This section describes how to configure ViewFs on the local HDFS cluster with HA setup to access the data on the remote HDFS HA cluster through Ambari.

Suppose there are two clusters, ClusterX and ClusterY. Each cluster is configured with NameNode HA. If applications running on ClusterX needs to access the data from ClusterY, then configure ViewFs on ClusterX. If applications running on ClusterY do not need access to the data on ClusterX, do not configure ViewFs on ClusterY. In the following example, we configure ViewFs on ClusterX and all configurations are done through ClusterX’s Ambari GUI.

Setup instructions

  1. Go to Ambari > HDFS > CONFIGS > ADVANCED > Advanced core-site and change the fs.defaultFS from hdfs://<fs_name> to viewfs://<federation_name>.

    For example, the property fs.defaultFS field is changed to viewfs://federationcluster.

  2. Go to Ambari > HDFS > CONFIGS > Custom hdfs-site and add or change all the properties as shown below.
    Note: Configure the key=value pair properties based on your cluster information.
    Check/modify existing Cluster X information
-----------------------------------------------
dfs.client.failover.proxy.provider.ClusterX=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.namenodes.ClusterX=nn1,nn2
dfs.internal.nameservices=ClusterX
dfs.namenode.http-address.ClusterX.nn1=c902f14x01.gpfs.net:50070
dfs.namenode.http-address.ClusterX.nn2=c902f14x03.gpfs.net:50070
dfs.namenode.rpc-address.ClusterX.nn1=c902f14x01.gpfs.net:8020
dfs.namenode.rpc-address.ClusterX.nn2=c902f14x03.gpfs.net:8020
dfs.nameservices=ClusterX, ClusterY

ADD new ClusterY key=value pair information
----------------------------------------------
dfs.client.failover.proxy.provider.ClusterY=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.ha.namenodes.ClusterY=nn1,nn2
dfs.namenode.http-address.ClusterY.nn1=c902f14x04.gpfs.net:50070
dfs.namenode.http-address.ClusterY.nn2=c902f14x06.gpfs.net:50070
dfs.namenode.rpc-address.ClusterY.nn1=c902f14x04.gpfs.net:8020
dfs.namenode.rpc-address.ClusterY.nn2=c902f14x06.gpfs.net:8020
  3. Go to Ambari > HDFS > CONFIGS > ADVANCED > Advanced viewfs-mount-table and specify the ViewFs mount table entries. You must define the mount table entries to map to the physical locations to the corresponding mount points from the ViewFs cluster. You must consider factors such as data access, mount levels, and application requirements while defining ViewFs mount table entries.
    Note: Place the <configuration> </configuration> entries with the corresponding <property></property> values based on your environment into the Advanced views-mount-table field using the xml format.

    For example, in ClusterX add the following properties into the viewfs-mount-table tab:

    <configuration>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./app-logs</name>
    <value>hdfs://ClusterX/app-logs</value>
  </property>
  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./apps</name>
    <value>hdfs://ClusterX/apps</value>
  </property>
  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./ats</name>
    <value>hdfs://ClusterX/ats</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./atsv2</name>
    <value>hdfs://ClusterX/atsv2</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./hdp</name>
    <value>hdfs://ClusterX/hdp</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./livy2-recovery</name>
    <value>hdfs://ClusterX/livy2-recovery</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./mapred</name>
    <value>hdfs://ClusterX/mapred</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./mr-history</name>
    <value>hdfs://ClusterX/mr-history</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./services</name>
    <value>hdfs://ClusterX/services</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./spark2-history</name>
    <value>hdfs://ClusterX/spark2-history</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./tmp</name>
    <value>hdfs://ClusterX/tmp</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./user</name>
    <value>hdfs://ClusterX/user</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./warehouse</name>
    <value>hdfs://ClusterX/warehouse</value>
  </property>

  <property>
    <name>fs.viewfs.mounttable.federationcluster.link./gpfs</name>
    <value>hdfs://ClusterY/gpfs</value>
  </property>

</configuration>
  4. Save the configuration and restart all the required services.