Manual HA switch configuration

High Availability (HA) is implemented in HDFS Transparency by using a shared directory in the IBM Storage Scale file system.

If your HDFS Transparency version is 2.7.0-2+, configure automatic HA according to Automatic NameNode service HA.

In the following configuration example, the HDFS nameservice ID is mycluster and the NameNode IDs are nn1 and nn2.

Define the nameservice ID in the core-site.xml file that is used by the Hadoop distribution. If you are using IBM® BigInsights® IOP or Hortonworks HDP, change this configuration in the Ambari GUI and restart the HDFS services to synchronize it with all the Hadoop nodes.
```
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluster</value>
</property>
```
Configure the hdfs-site.xml file that is used by the Hadoop distro. If you are using IBM BigInsights IOP or Hortonworks HDP, change these configurations in the Ambari GUI and restart the HDFS services to synchronize it with all the Hadoop nodes.
```
<property>

<name>dfs.nameservices</name>
<value>mycluster</value>
</property>
```
```
<property>

<name>dfs.ha.namenodes.mycluster</name>
<value>nn1,nn2</value>
</property>
```
```
<property>

<name>dfs.namenode.rpc-address.mycluster.nn1</name>
<value>c8f2n06.gpfs.net:8020</value>
</property>
```
```
<property>

<name>dfs.namenode.rpc-address.mycluster.nn2</name>
<value>c8f2n07.gpfs.net:8020</value>
</property>
```
```
<property>

<name>dfs.namenode.http-address.mycluster.nn1</name>
<value>c8f2n06.gpfs.net:50070</value>
</property>
```
```
<property>

<name>dfs.namenode.http-address.mycluster.nn2</name>
<value>c8f2n07.gpfs.net:50070</value>
</property>
```
```
<property>


<name>dfs.namenode.shared.edits.dir</name>
<value>file:///<gpfs.mnt.dir>/HA</value>
</property>
```
```
<property>
<name>dfs.ha.standby.checkpoints</name>
<value>false</value>
</property>
```
```
<property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
```
The native HDFS uses the directory specified by dfs.namenode.shared.edits.dir configuration parameter to save information shared between the active NameNode and the standby NameNode. HDFS Transparency 2.7.3-4 and 3.1.0-0 uses this directory to save Kerberos delegation token related information. If you revert from HDFS Transparency back to the native HDFS, please revert dfs.namenode.shared.edits.dir configuration parameter back to the one used for the native HDFS. In Ambari Mpack 2.4.2.7 and Mpack 2.7.0.1, the dfs.namenode.shared.edits.dir parameter is set automatically when integrating or unintegrating IBM Storage Scale service.

The dfs.namenode.shared.edits.dir configuration parameter must be consistent with gpfs.mnt.dir defined in /usr/lpp/mmfs/hadoop/etc/hadoop/gpfs-site.xml. You can create the directory /<gpfs.mnt.dir>/HA and change the ownership to hdfs:hadoop before starting the HDFS transparency services.

For HDFS Transparency releases earlier than 2.7.3-4 and also the 3.0.0 release, the dfs.ha.standby.checkpoints must be set as false. If not, you will see a log of exceptions in the standby NameNode logs. However, for HDFS Transparency 2.7.3-4 and 3.1.0-0, the dfs.ha.standby.checkpoints must be set as true.
For example,
```
ERROR ha.StandbyCheckpointer (StandbyCheckpointer.java:doWork(371)) - Exception in doCheckpoint
```
If you are using Ambari, search under HDFS > Configs to see if the dfs.ha.standby.checkpoints field is set to false. If the field is not found, add the field to the Custom hdfs-site > Add property and set it to false. Save Config, and restart HDFS and any services that you might see with restart icon in Ambari.
Note: This disables the exception written to the log.

To not show the alert from Ambari, click on the alert and disable the alert from Ambari GUI.

HDFS transparency does not have fsImage and editLogs. Therefore, do not perform checkpoints from the standby NameNode service.

After 2.7.3-4 of 2.x and 3.1.0-0 of 3.x, HDFS transparency has minimal EditLogs support. Therefore, need checkpoints (dfs.ha.standby.checkpoints=true setting) to clean up outdated EditLogs. The 3.0.0 release of HDFS transparency does not support EditLog (dfs.ha.standby.checkpoints=false setting).

The dfs.client.failover.proxy.provider.mycluster configuration parameter must be changed according to the name service ID. In the above example, the name service ID is configured as mycluster in core-site.xml. Therefore, the configuration name is dfs.client.failover.proxy.provider.mycluster.
Note: If you enable Short Circuit Read in the Short Circuit Read Configuration section, the value of the configuration parameter must be org.apache.hadoop.gpfs.server.namenode.ha.ConfiguredFailoverProxyProvider.
Follow the guide in the Sync Hadoop configurations section to synchronize core-site.xml and hdfs-site.xml from the Hadoop distribution to any one node that is running HDFS transparency services. For example, HDFS_Transparency_node1.
For HDFS Transparency 2.7.0-x, on HDFS_Transparency_node1, modify /usr/lpp/mmfs/hadoop/etc/hadoop/hdfs-site.xml:
```
 <property>
<name>dfs.client.failover.proxy.provider.mycluster</name>
<value>org.apache.hadoop.gpfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
```
With this configuration, WebHDFS service functions correctly when NameNode HA is enabled.
Note: On HDFS transparency nodes, the configuration value of the key dfs.client.failover.proxy.provider.mycluster in hdfs-site.xml is different from that in Step2.

Note: This step should not be performed from HDFS Transparency 2.7.2-x.
On HDFS_Transparency_node1, run the command as the root user to synchronize the HDFS Transparency configuration to all the HDFS transparency nodes:
# mmhadoopctl connector syncconf /usr/lpp/mmfs/hadoop/etc/hadoop
If you are using HDFS Transparency version 2.7.3-4 or 3.1.0-0, before you start HDFS transparency for the first time, run the /usr/lpp/mmfs/hadoop/bin/hdfs namenode -initializeSharedEdits command to initialize the shared edits directory. Run this command only after configuring HA and before starting the service.
Start the HDFS transparency service by running the mmhadoopctl command:
# mmhadoopctl connector start
After the service starts, both NameNodes are in the standby mode by default. You can activate one NameNode by using the following command so that it responds to the client:
# /usr/lpp/mmfs/hadoop/bin/gpfs haadmin -transitionToActive --forceactive [namenode ID]
For example, you can activate the nn1 NameNode by running the following command:
# /usr/lpp/mmfs/hadoop/bin/gpfs haadmin -transitionToActive -forceactive nn1
If the nn1 NameNode fails, you can activate another NameNode and relay the service by running the following command:
# /usr/lpp/mmfs/hadoop/bin/gpfs haadmin -transitionToActive -forceactive nn2

Note: The switch must be done manually. Automatic switch will be supported in the future releases.
Use the following command to view the status of the NameNode:
# /usr/lpp/mmfs/hadoop/bin/gpfs haadmin -getServiceState [namenode ID]
You could check your /usr/lpp/mmfs/hadoop/etc/hadoop/hdfs-site.xml or run the following commands to figure out the [NameNode ID]:
```
#/usr/lpp/mmfs/hadoop/bin/gpfs getconf -confKey fs.defaultFS
hdfs://mycluster
#hdfs getconf -confKey dfs.ha.namenodes.mycluster
nn1,nn2
```
After one NameNode becomes active, you can start the other Hadoop components, such as hbase and hive and run your Hadoop jobs.
Note: When HA is enabled for HDFS transparency, you might see the following exception in the logs: Get corrupt file blocks returned error: Operation category READ is not supported in state standby.

These are known HDFS issues: HDFS-3447 and HDFS-8910.