Manually enabling TLS for HDFS Transparency

This section lists the steps to manually enable TLS for HDFS Transparency.

Overview:

For enabling TLS on the CES HDFS Transparency cluster, the following must be created on the CES HDFS nodes:

Truststores
Certificates
Keystores

You must then verify that the IBM Storage Scale HDFS clients can securely access the IBM Storage Scale file system.

Before following these steps, see the Prerequisites topic.

Procedure:

On all the HDFS Transparency nodes, as root, create a directory /etc/security/serverKeys/ where the TLS keys and certificates can be stored. On all the HDFS Transparency nodes, run the following command:
```
# mkdir -p /etc/security/serverKeys/
```

Log in to one of the CES HDFS NameNode as root (for example, NameNode1) and run all the following commands from that one node:

Create a keystore specific to CES HDFS IP, using the keytool -genkey command.

# keytool -genkey -alias <CES_HOSTNAME_FQDN> -keyalg RSA -keysize 2048 -validity 1800 -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks

where, <CES_HOSTNAME_FQDN> is the FQDN Hostname corresponding to the CES IP configured for your CES HDFS cluster.

After you run the command, enter a password for the keystore, a password for the key, your first and last name, and your organization and location details.

Note:

Your first name and last name must be same as your CES IP hostname.
The keystore password and the key password will be needed for later configuration steps. Therefore, keep the passwords in a safe place.

For example,

# keytool -genkey -alias cesip09x15.gpfs.net -keyalg RSA -keysize 2048 -validity 1800 -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks

Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  cesip09x15.gpfs.net
What is the name of your organizational unit?
  [Unknown]:  IBM
What is the name of your organization?
  [Unknown]:  IBM
What is the name of your City or Locality?
  [Unknown]:  Poughkeepsie
What is the name of your State or Province?
  [Unknown]:  New York
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=cesip09x15.gpfs.net, OU=IBM, O=IBM, L=Poughkeepsie, ST=New York, C=US correct?
  [no]:  yes

Enter key password for <cesip09x15.gpfs.net>
        (RETURN if same as keystore password):

Warning:
The JKS keystore uses a proprietary format. It is recommended to migrate to PKCS12 which is an industry standard format using "keytool -importkeystore -srckeystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -destkeystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -deststoretype pkcs12".
#

For the keystore that is created in the above step, export the certificate public key to a certificate file.

# keytool -export -alias <CES_HOSTNAME_FQDN> -keystore  /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -rfc -file  /etc/security/serverKeys/<CES_HOSTNAME_FQDN>.pem

where, <CES_HOSTNAME_FQDN> is the FQDN hostname corresponding to the CES IP configured for your CES HDFS cluster.

For example,

# keytool -export -alias cesip09x15.gpfs.net -keystore  /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -rfc -file  /etc/security/serverKeys/cesip09x15.gpfs.net.pem
Enter keystore password:
Certificate stored in file </etc/security/serverKeys/cesip09x15.gpfs.net.pem

Distribute the generated Keystore to all the HDFS Transparency NameNodes. This keystores will be updated in the next step.
Run the following command for each <HDFS Transparency NameNode>:
```
# scp /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks  root@<HDFS Transparency NameNode>:/etc/security/serverKeys/
```
Note: Do not copy the Keystore to DataNodes. DataNodes have their own Keystores that will be created in step 4.

On each CES HDFS NameNode, run the following commands as a root user:
1. Update the keystore specific to this NameNode hostname.
```
# keytool -genkey -alias <NN_FQDN_HOSTNAME> -keyalg RSA -keysize 2048 -validity 1800 -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks
```
  where, <NN_FQDN_HOSTNAME> is the FQDN hostname of this NameNode. When you are prompted, use the keystore password from step 2.a. This command will append the generated keystore to the already existing keystore created specific to the CES HDFS IP hostname.
2. For the keystore that is created in the above step, export the certificate public key to a certificate file.
```
# keytool -export -alias <NN_FQDN_HOSTNAME> -keystore  /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -rfc -file  /etc/security/serverKeys/<NN_FQDN_HOSTNAME>.pem
```
  where, <NN_FQDN_HOSTNAME> is the FQDN hostname of this NameNode. When you are prompted, use the keystore password from step 2.a.
On each HDFS Transparency DataNode, run the following commands as a root user:
1. Create a keystore specific to this DataNode.
```
# keytool -genkey -alias <DN_HOSTNAME_FQDN> -keyalg RSA -keysize 2048 -validity 1800 -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks
```
  where, <DN_FQDN_HOSTNAME> is the FQDN hostname of this DataNode. When prompted, assign a password for the keystore.
2. For the keystore that is created in the above step, export the certificate public key to a certificate file specific to this DataNode.
```
# keytool -export -alias <DN_HOSTNAME_FQDN> -keystore  /etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks -rfc -file  /etc/security/serverKeys/<DN_HOSTNAME_FQDN>.pem
```
  where, <DN_FQDN_HOSTNAME> is the FQDN Hostname of this DataNode.
  
  When you are prompted, use the keystore password from step 4.a.
Create a Master Truststore for HDFS Transparency. Log into the same CES HDFS NameNode as in step 2 and run the following commands from that node:
1. Create a /etc/security/serverKeys/trust_store/ directory where the created Truststore .jks files corresponding to all NameNodes, DataNodes, and the CES IP will be merged together to create a master Truststore.
```
# mkdir -p /etc/security/serverKeys/trust_store/
```
2. From the NameNode and DataNodes hosts, copy all the .pem files from the /etc/security/serverKeys/ directory to the /etc/security/serverKeys/trust_store/ directory on this NameNode.
  For each <HDFS Transparency node>, run the following command:
```
# scp <HDFS Transparency node>:/etc/security/serverKeys/*.pem  /etc/security/serverKeys/trust_store/
```
  Ensure that all the .pem files corresponding to every NameNode, DataNode hostname as well as the CES IP hostname are copied over to /etc/security/serverKeys/trust_store/.
3. For each <.pem file> in the /etc/security/serverKeys/trust_store/ directory, run the following command:
```
# keytool -import -noprompt -alias <FQDN Hostname Corresponding the .pem file> -file /etc/security/serverKeys/trust_store/<.pem file> -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks
```
  This will import the certificates into a Master Truststore /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks for HDFS Transparency:
  
  For example,
  If namenode1.gpfs.net.pem is the certificate file corresponding to the host namenode1.gpfs.net, run the following command:
```
# keytool -import -noprompt -alias namenode1.gpfs.net -file /etc/security/serverKeys/trust_store/namenode1.gpfs.net.pem  -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks
```
  Note: When you import the first .pem file, you need to set a password for the Master Truststore. Use this password for importing the remaining .pem files.
4. Repeat the keytool -import command for all the .pem files.
Distribute the master Truststore and master certificate to all the HDFS Transparency nodes (NameNodes and DataNodes).
Run the following command specific to each <HDFS Transparency node>:
```
# scp /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks  root@<HDFS Transparency node>:/etc/security/serverKeys/
```

Create and update the CES HDFS Transparency configuration files.

Create a directory to host the configuration files. This directory will host existing and newly created configuration files.
```
# mkdir /tmp/hdfsconf
```

Get the existing configuration files from CCR into that directory.

# /usr/lpp/mmfs/hadoop/sbin/mmhdfs config export /tmp/hdfsconf core-site.xml,hdfs-site.xml,ssl-server.xml,ssl-client.xml

Update the existing config files with the following changes based on your environment:

File: core-site.xml

<property>
    <name>hadoop.ssl.require.client.cert</name>
    <value>false</value>
  </property>
  
<property>
    <name>hadoop.ssl.hostname.verifier</name>
    <value>DEFAULT</value>
  </property>

  <property>
    <name>hadoop.ssl.keystores.factory.class</name>
    <value>org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory</value>
  </property>
 
 <property>
    <name>hadoop.ssl.server.conf</name>
    <value>ssl-server.xml</value>
  </property>
  
<property>
    <name>hadoop.ssl.client.conf</name>
    <value>ssl-client.xml</value>
  </property>

File: hdfs-site.xml

<property>
    <name>dfs.http.policy</name>
    <value>HTTPS_ONLY</value>
  </property>
  <property>
    <name>dfs.client.https.need-auth</name>
    <value>false</value>
  </property>
  <property>
    <name>dfs.namenode.https-bind-host</name>
    <value>0.0.0.0</value>
  </property>
<property>
   <name>dfs.namenode.https-address.<cluster name>.nn1</name>
   <value><NameNode 1 hostname>:50470</value>
</property>
<property>
   <name>dfs.namenode.https-address.<cluster name>.nn2</name>
   <value><NameNode 2 hostname>:50470</value>
</property>

where,

<NameNode 1 hostname> and <NameNode 2 hostname> are the actual CES HDFS NameNode FQDN hostnames.

<cluster name> is the name of your CES HDFS cluster that is also your HDFS Namespace.

If you want both the secure and unsecure http connections, set dfs.http.policy to HTTP_AND _HTTPS.

File: ssl-server.xml

<property>
    <name>ssl.server.truststore.location</name>
    <value>/etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks</value> 
  </property>
  <property>
    <name>ssl.server.truststore.password</name>
    <value><truststore_password></value>
  </property>
  <property>
    <name>ssl.server.truststore.type</name>
    <value>jks</value>
  </property>
  <property>
    <name>ssl.server.truststore.reload.interval</name>
    <value>10000</value>
  </property>
  <property>
    <name>ssl.server.keystore.location</name>
    <value>/etc/security/serverKeys/spectrum_scale_ces_hdfs_keystore.jks</value>
  </property>
  <property>
    <name>ssl.server.keystore.password</name>
    <value><keystore_password></value>
  </property>
  <property>
    <name>ssl.server.keystore.keypassword</name>
    <value><key_password></value>
  </property>
  <property>
    <name>ssl.server.keystore.type</name>
    <value>jks</value>
  </property>

where,

<keystore_password> and <key_password> are the corresponding actual passwords from 2.a.

<truststore_password> is the corresponding actual password from 5.c.

File: ssl-client.xml

<property>
    <name>ssl.client.truststore.location</name>
    <value>/etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks</value> 
  </property>

  </property>
  <property>
    <name>ssl.client.truststore.password</name>
    <value><truststore_password></value>
  </property>
  <property>
    <name>ssl.client.truststore.type</name>
    <value>jks</value>
  </property>

where, <truststore_password> is the corresponding actual password from 5.c.

Update the CES HDFS configuration to IBM Storage Scale CCR repository and restart the HDFS Transparency services.
1. Stop HDFS Transparency services for the cluster.
  1. On any CES HDFS NameNode, run the following commands:
```
# mmhdfs hdfs-dn stop
# mmces service stop HDFS -N <NN1>,<NN2>
```
2. Import the existing and new configuration files to /var/mmfs/hadoop/etc/hadoop by running the following command:
```
# mmhdfs config import /tmp/hdfsconf core-site.xml,hdfs-site.xml,ssl-server.xml,ssl-client.xml
```
3. Upload the changes to CCR repository.
```
# mmhdfs config upload
```
4. Start the HDFS Transparency services for the cluster.
  1. On any CES HDFS NameNode, run the following command:
```
# mmhdfs hdfs-dn start
# mmces service start HDFS -N <NN1>,<NN2>
```
5. Verify that the CES HDFS services have started.
  On any CES HDFS NameNode, run the following command:
```
# mmhdfs hdfs status
```