Enable Kerberos

To enable Kerberos on the native HDFS cluster, the native HDFS cluster and the remote IBM Storage Scale HDFS Transparency cluster requires to have the same Kerberos principals for the HDFS service.

After setting up the local native HDFS cluster and the remote HDFS Transparency cluster based on the Common configuration section, following these additional steps to configure Kerberos:

Enable Kerberos on the local native HDFS/HDP cluster by installing a new MIT KDC by following the Hortonworks documentation for Configuring Ambari and Hadoop for Kerberos.
Perform the following configuration changes on the remote HDFS Transparency cluster:
For cluster with Ambari:
1. Follow the Setting up KDC server and enabling Kerberos, using the MIT KDC server already setup in the above so as to manage the same test user account (such as hdp-user1 in below examples) Principal/Keytab on both local native HDFS cluster and remote IBM Storage Scale HDFS Transparency cluster.
2. By default, HDP configures the service principals followed by the cluster name. If the remote HDFS Transparency cluster has the same cluster name as the local native HDFS/HDP cluster, the default principal values on either of these clusters need to manually configure its Kerberos principals to have a different service principal name.
  For example, if the remote HDFS Transparency cluster’s cluster name is REMOTE, the default principal for the remote HDFS Transparency HDFS service is set as hdfs-REMOTE@{REALMNAME}. If the local native HDFS/HDP cluster also have the same cluster name (REMOTE), the HDFS service on the native HDFS/HDP cluster will fail to start.
  
  If you do not change the remote HDFS Transparency cluster service principals, one can change the local native HDFS/HDP cluster default service principal to another value like hdfs@{REALMNAME}.
3. If the remote HDFS Transparency cluster has a different cluster name than the local native HDFS/HDP cluster, Kerberos can be enabled by following the Setting up KDC server and enabling Kerberos section. After enabling Kerberos, add all the service principal rules from the local native cluster to the remote HDFS Transparency cluster.
  For example, go to Ambari > Services > HDFS > CONFIGS > Advanced core-site > hadoop.security.auth_to_local to find all the service principals and copy them to the remote Transparency cluster.
```
RULE:[1:$1@$0](accumulo-hdptest@IBM.COM)s/.*/accumulo/
RULE:[1:$1@$0](ambari-qa-hdptest@IBM.COM)s/.*/ambari-qa/
RULE:[1:$1@$0](druid-hdptest@IBM.COM)s/.*/druid/
RULE:[1:$1@$0](hbase-hdptest@IBM.COM)s/.*/hbase/
RULE:[1:$1@$0](hdfs-hdptest@IBM.COM)s/.*/hdfs/
RULE:[1:$1@$0](spark-hdptest@IBM.COM)s/.*/spark/
RULE:[1:$1@$0](tracer-hdptest@IBM.COM)s/.*/accumulo/
RULE:[1:$1@$0](yarn-ats-hdptest@IBM.COM)s/.*/yarn-ats/
RULE:[1:$1@$0](zeppelin-hdptest@IBM.COM)s/.*/zeppelin/
```
For cluster without Ambari:

Note: From HDFS Transparency version 3.x, the HDFS Transparency configuration directory is changed from /usr/lpp/mmfs/hadoop/etc/hadoop to /var/mmfs/hadoop/etc/hadoop. Ensure that the correct directory paths are used with the corresponding changes when manually configuring HDFS Transparency.
1. Do not copy the hadoop-env.sh from the local native HDFS/HDP cluster to the HDFS Transparency cluster.
2. If dfs.client.read.shortcircuit is true, run the following command on one of the HDFS Transparency nodes. Otherwise, the HDFS Transparency DataNode fails to start.
```
/usr/lpp/mmfs/bin/mmdsh -N all “chown root:root -R /var/lib/hadoop-hdfs”
```
  No change is required on the HDFS Transparency cluster if the dfs.client.read.shortcircuit is set to false in the hdfs-site.xml on the local native HDFS cluster.
3. Copy the configuration files, core-site.xml and hdfs-site.xml, located in /etc/hadoop/conf from the local native HDFS cluster to /usr/lpp/mmfs/hadoop/etc/hadoop on one of node from the HDFS Transparency cluster.
4. Change the NameNode value from the local native HDFS cluster NameNode to the HDFS Transparency NameNode on the HDFS Transparency node selected in 2.c for both the core-site.xml and hdfs-site.xml files.
5. Remove the property net.topology.script.file.name in /usr/lpp/mmfs/hadoop/etc/hadoop/core-site.xml and remove the property dfs.hosts.exclude and secondary NameNode related properties dfs.namenode.secondary.http-address, dfs.namenode.checkpoint.dir, dfs.secondary.namenode.kerberos.internal.spnego.principal, dfs.secondary.namenode.kerberos.principal, dfs.secondary.namenode.keytab.file in /usr/lpp/mmfs/hadoop/etc/hadoop/hdfs-site.xml on the HDFS Transparency node selected in 2.c.
6. On the HDFS Transparency node selected in 2.c, run /usr/lpp/mmfs/bin/mmhadoopctl connector syncconf /usr/lpp/mmfs/hadoop/etc/hadoop to sync all these changes into the other HDFS Transparency nodes.

Enable Kerberos on the remote HDFS Transparency cluster.

For cluster with Ambari

Follow the Enabling Kerberos when the IBM Spectrum Scale service is integrated to enable Kerberos on IBM Storage Scale HDFS Transparency cluster.

For cluster without Ambari:

Ensure the HDFS Transparency cluster is not in running status.
```
/usr/lpp/mmfs/bin/mmhadoopctl connector status
```
Using the same KDC server with the local native HDFS/HDP cluster.
Install the Kerberos clients package on all the HDFS Transparency nodes.
```
yum install -y krb5-libs krb5-workstation
```

Sync the KDC Server config, /etc/krb5.conf, to the Kerberos clients (All the HDFS Transparency nodes).

HDFS Transparency principals and keytabs list information:

Component	Principal name	Keytab File Name
NameNode	nn/$NN_Host_FQDN@REALMS	nn.service.keytab
NameNode HTTP	HTTP/$NN_Host_FQDN@REALMS	spnego.service.keytab
DataNode	dn/$DN_Host_FQDN@REALMS	dn.service.keytab

Note: Replace the NN_Host_FQDN with your HDFS Transparency NameNode hostname and replace the DN_Host_FQDN with your HDFS Transparency DataNode hostname. If HDFS Transparency NameNode HA is configured, you need to have two principals for both NameNodes. It is required to have one principal for each HDFS Transparency DataNode.

Add the principals above to the Kerberos database on the KDC Server.
```
#kadmin.local
#kadmin.local:  add_principal -randkey nn/$NN_Host_FQDN@REALMS
#kadmin.local:  add_principal -randkey HTTP/$NN_Host_FQDN@REALMS
#kadmin.local:  add_principal -randkey dn/$DN_Host_FQDN@REALMS
```
Note: Replace the NN_Host_FQDN and DN_Host_FQDN with your cluster information. It is required to have one principal for each HDFS Transparency DataNode.
Create a directory for the keytab directory and set the appropriate permissions on each of the HDFS Transparency node.
```
mkdir -p /etc/security/keytabs/
chown root:root /etc/security/keytabs
chmod 755 /etc/security/keytabs
```

Generate the keytabs for the principals.

#xst -norandkey -k /etc/security/keytabs/nn.service.keytab  nn/$NN_Host_FQDN@REALMS

#xst -norandkey -k /etc/security/keytabs/spnego.service.keytab  HTTP/$NN_Host_FQDN@REALMS

#xst -norandkey -k /etc/security/keytabs/dn.service.keytab  dn/$DN_Host_FQDN@REALMS

Note: Replace the NN_Host_FQDN and DN_Host_FQDN with your cluster information. It is required to have one principal for each HDFS Transparency DataNode.

Copy the appropriate keytab file to each host. If a host runs more than one component (for example, both NameNode and DataNode), copy the keytabs for both components.

Set the appropriate permissions for the keytab files.

On the HDFS Transparency NameNode host(s):

chown root:hadoop /etc/security/keytabs/nn.service.keytab
chmod 400 /etc/security/keytabs/nn.service.keytab
chown root:hadoop /etc/security/keytabs/spnego.service.keytab 
chmod 440 /etc/security/keytabs/spnego.service.keytab

On the HDFS Transparency DataNode hosts:

chown root:hadoop /etc/security/keytabs/dn.service.keytab
chmod 400 /etc/security/keytabs/dn.service.keytab

Start the HDFS Transparency service from any one of the HDFS Transparency node with root passwordless ssh access to all the other HDFS Transparency nodes:
```
/usr/lpp/mmfs/bin/mmhadoopctl connector start
```

Validate the local native HDFS cluster when Kerberos is enabled by running a MapReduce wordcount workload.

Create user such as hdp-user1 and hdp-user2 on all the nodes of the local native HDFS cluster and the remote HDFS Transparency cluster (For example, c16f1n07.gpfs.net is the local native HDFS cluster NameNode, c16f1n03.gpfs.net is the remote HDFS Transparency cluster NameNode).
```
kinit -k -t /ect/security/keytabs/hdptestuser.headless.keytab hdp-user1@IBM.COM
```

The MapReduce wordcount workload by hdp-user1 and hdp-user2 will failed on the local native HDFS cluster node.

[root@c16f1n07 ~]# su hdp-user2
[hdp-user2@c16f1n07 root]$ klist
klist: Credentials cache file '/tmp/krb5cc_11016' not found
[hdp-user2@c16f1n07 root]$ yarn jar /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-examples.jar 
wordcount hdfs://c16f1n07.gpfs.net:8020/user/hdp-user1/redhat-release 
hdfs://c16f1n03.gpfs.net:8020/user/hdp-user1/redhat-release-wordcount
18/03/05 22:29:26 INFO client.RMProxy: Connecting to ResourceManager at c16f1n08.gpfs.net/192.0.2.1:8050
18/03/05 22:29:27 INFO client.AHSProxy: Connecting to Application History server at 
c16f1n08.gpfs.net/192.0.2.1:10200
18/03/05 22:29:27 WARN ipc.Client: Exception encountered while connecting to the server : 
javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials 
provided (Mechanism level: Failed to find any Kerberos tgt)]
java.io.IOException: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: 
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: 
Failed to find any Kerberos tgt)]; Host Details : local host is: "c16f1n07/192.0.2.0"; 
destination host is: "c16f1n03.gpfs.net":8020; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:785)
	at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1558)
	at org.apache.hadoop.ipc.Client.call(Client.java:1498)
	at org.apache.hadoop.ipc.Client.call(Client.java:1398)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
	at com.sun.proxy.$Proxy10.getDelegationToken(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getDelegationToken
(ClientNamenodeProtocolTranslatorPB.java:985)

To fix the MapReduce wordcount workload error, generate the principal and keytab for user hdp-user1 on the KDC server.

# kadmin.local
#kadmin.local:  add_principal -randkey hdp-user1
WARNING: no policy specified for hdp-user1@IBM.COM; defaulting to no policy
Principal "hdp-user1@IBM.COM" created.
kadmin.local:  xst -norandkey -k /etc/security/keytabs/hdptestuser.headless.keytab hdp-user1@IBM.COM
Entry for principal hdp-user1@IBM.COM with kvno 1, encryption type aes256-cts-hmac-sha1-96 
added to keytab WRFILE:/etc/security/keytabs/hdptestuser.headless.keytab.
Entry for principal hdp-user1@IBM.COM with kvno 1, encryption type aes128-cts-hmac-sha1-96 
added to keytab WRFILE:/etc/security/keytabs/hdptestuser.headless.keytab.
Entry for principal hdp-user1@IBM.COM with kvno 1, encryption type des3-cbc-sha1 added to 
keytab WRFILE:/etc/security/keytabs/hdptestuser.headless.keytab.
Entry for principal hdp-user1@IBM.COM with kvno 1, encryption type arcfour-hmac added to 
keytab WRFILE:/etc/security/keytabs/hdptestuser.headless.keytab.
kadmin.local:

Copy the hdp-user1 keytab to all the nodes of the local native HDFS cluster and the remote HDFS Transparency cluster and change the permission for the hdp-user1 keytab file.

[root@c16f1n07 keytabs]#pwd
/etc/security/keytabs
[root@c16f1n07 keytabs]# chown hdp-user1 /etc/security/keytabs/hdptestuser.headless.keytab
[root@c16f1n07 keytabs]# chmod 400 /etc/security/keytabs/hdptestuser.headless.keytab

Re-run the MapReduce wordcount workload by user hdp-user1 to ensure that no errors are seen.