Enabling NameNode HA

This topic lists the steps to enable NameNode HA within CDP Private Cloud Base with IBM Storage Scale service in Cloudera Manager.

Note: Because the compute and storage architecture are decoupled, the server-side administration of NameNode HA is managed by the IBM Storage Scale CES protocol. Unlike native HDFS, Zookeeper/zkfc is not used for IBM Storage Scale NameNode HA.
The following are the two steps to the HA enablement process:
  • Server side: NameNode HA can be enabled in the CES HDFS cluster during the installation and deployment using the IBM Storage Scale installation toolkit. However, if NameNode HA is not enabled on your CES HDFS cluster, follow Change CES HDFS NON-HA cluster into CES HDFS HA cluster to enable it.
  • Client side: Now enable the NameNode HA for the IBM Storage Scale service in the Cloudera Manager by enabling the NameNode HA for the CDP Private Cloud Base cluster. This means that when a NameNode failover event occurs in the IBM Storage Scale CES HDFS cluster, HDFS clients and Hadoop workloads running on the CDP Private Cloud Base cluster retry to connect similar native HDFS HA environments.
In the following procedure, the HDFS Transparency cluster name is <cluster-name> and the hostname corresponding to CES IP configured on HDFS Transparency is <myceshost>:
  1. From the Cloudera Manager GUI, stop all services.
  2. In the Cloudera Manager GUI, modify default_fs_name (Default File System URL) from hdfs://<myceshost>:8020 to hdfs://<cluster-name>.
  3. In the Cloudera Manager GUI, add the following configurations:
    • Click SpectrumScale > Actions Configuration > Cluster-wide Advanced Configuration Snippet (Safety Valve) for core-site.xml and set the custom parameter to the following value:

      fs.defaultFS to hdfs://<cluster-name>

    • Click Spectrum Scale > Configuration > Transparency NameNode Advanced Configuration Snippet (Safety Valve) for core-site.xml and set the same custom fs.defaultFS parameter to the following value:

      fs.defaultFS to hdfs://<cluster-name>

    • Click SpectrumScale > Configuration > HDFS Client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml and set the custom parameters to the following values:
      • dfs.nameservices to <cluster-name>
      • dfs.ha.namenodes.<cluster-name> to nn1
      • dfs.namenode.rpc-address.<cluster-name>.nn1 to <myceshost>:8020
      • dfs.namenode.http-address.<cluster-name>.nn1 to <myceshost>:50070
      • dfs.client.failover.proxy.provider.<cluster-name> to org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
      • If the CES HDFS cluster is NameNode HA enabled, set dfs.namenode.https-address.<cluster-name>.nn1 to <myceshost>:50470.
      • Under SpectrumScale > Configuration, search for webhdfs_url parameter and set the value to blank.
      Note: 8020, 50070 and 50470 are the default RPC, HTTP and HTTPS ports for NameNode. If you are not using these default ports, you must update the parameters accordingly.
  4. Save the changes.
  5. Restart the services with stale client configuration.
  6. To view the NameNode Active/Standby states using IBM Storage Scale CLI, see Monitoring NameNodes and to view the NameNode Active/Standby states using Cloudera Manager, see Monitoring.