Enabling TLS

Cloudera Manager does not manage TLS for the IBM Storage Scale service because CDP Private Cloud Base cluster and CES HDFS cluster are loosely integrated, and are considered different clusters even if the DataNodes are colocated with the CDP Private Cloud Base nodes. Therefore, TLS needs to be manually enabled on the CES HDFS cluster. Also, verify that TLS is working on CES HDFS before proceeding to enable TLS on the Cloudera Manager.

Enable auto-TLS on Cloudera Manager before creating CDP Private Cloud Base clusters. This enables TLS for the rest of the CDP Private Cloud Base services (for example, YARN, Zookeeper, Hive, etc). However, enabling auto-TLS does not automatically enable TLS for IBM Storage Scale. Therefore, TLS for IBM Storage Scale must first be enabled independently of Cloudera.

If the CDP Private Cloud Base cluster with IBM Storage Scale is already created, proceed to the next step. Otherwise, create it now, by following the instructions in Installing Cloudera Data Platform Private Cloud Base with IBM Storage Scale.

In order to update the certificates, all the services in Cloudera and in IBM Storage Scale CES HDFS Transparency need to be stopped.

Click on Action > Deploy Client configuration from the main cluster view in Cloudera Manager. This will propagate ssl-client.xml and the other configuration files to under /etc/hadoop/conf. The ssl-client.xml file will be referenced under this step below.

Then exchange Cloudera and IBM Storage Scale certificates to each other's truststore. There are two options to do so, either by using the provided automation or by using the manual procedure.

1. Using Automation -

   From IBM Storage Scale 5.1.1.1 and HDFS Transparency 3.1.1-5, you can use the gpfs_tls_configuration.py script to update Cloudera Manager and the CES HDFS certificates in each other's trust store so that Cloudera services can work properly with HDFS Transparency in a TLS-enabled environment. This can be done using the following automation script, or you can import the certificates manually as mentioned in the following step:

   The gpfs_tls_configuration.py script can be used to exchange the certificates.

   This script performs following steps:

   1. Imports the Cloudera Manager public certificates to IBM Storage Scale trust store.
   2. Imports the IBM Storage Scale certificates to the Cloudera Manager trust store.
   Run the following command to exchange the certificates from one of the CES NameNodes:

   /usr/lpp/mmfs/hadoop/scripts/gpfs_tls_configuration.py integrate-with-cdp CDP-TRUSTSTORE-PASSWORD-FILE

   where, CDP-TRUSTSTORE-PASSWORD-FILE is the JSON file containing the cdp-truststore-password.

   {
     "cdp-truststore-password": "PASSWORD"
   }

   The CDP trust store password is located in /etc/hadoop/conf/ssl-client.xml under the ssl.client.truststore.password parameter.

   Note: After the script has run successfully, the password file will be automatically deleted for security reason.
   For example:

     /usr/lpp/mmfs/hadoop/scripts/gpfs_tls_configuration.py integrate-with-cdp /tmp/cdppassword.json
     [ INFO  ] Importing Cloudera Manager public certificate (cm-auto-global_cacerts.pem) to /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks
     [ INFO  ] Copying updated /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks to /var/lib/cloudera-scm-agent/agent-cert/
     [ INFO  ] Distributing /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks to all the Transparency nodes
     [ INFO  ] Adding all scale   nodes certificates to cm-auto-global_cacerts.pem
     [ INFO ] NOTE: You will need to manually distribute the following files to all the CDP nodes :
               1. /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
               2. /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks
     
   
   

   Distribute the following files to all the CDP nodes:

     1. /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem
     2. /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks
   

2. Using Manual procedure -

   For IBM Storage Scale 5.1.1.0 and HDFS Transparency 3.1.1-4:

   Update Cloudera Manager and CES HDFS certificates in each other's trust store so that Cloudera services can work properly with HDFS Transparency in a TLS enabled environment.

   Log into the CES HDFS NameNode containing the HDFS Transparency certificate (.pem) files. To find that NameNode, see step 2 in Manually enabling TLS for HDFS Transparency.

   Log into that NameNode and run the following commands:

   • Add Cloudera Manager global CA certificate to CES HDFS Truststore spectrum_scale_ces_hdfs_truststore.jks.

     # keytool -noprompt -importcert -alias cloudera-agents -file /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem -keystore /etc/security/serverKeys/spectrum_scale_ces_hdfs_truststore.jks

     This command prompts the user to enter the CES HDFS Master Truststore password. To find the Truststore password, see step 5.c of Manually enabling TLS for HDFS Transparency.

   • Append the HDFS Transparency certificates to Cloudera global CA certificate agent-cert/cm-auto-global_cacerts.pem by running the following command:

     # cat /etc/security/serverKeys/trust_stores/*.pem >> /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_cacerts.pem

   • Import the HDFS Transparency certificates to Cloudera global Truststore cm-auto-global_truststore.jks.
     For each <.pem file> in the /etc/security/serverKeys/trust_store/ directory, run the following command:

     keytool -noprompt -importcert -alias <FQDN Hostname Corresponding the .pem file> -file /etc/security/serverKeys/trust_store/<name of .pem file> -keystore /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks

     For example,

     If namenode1.gpfs.net.pem is the certificate file corresponding to the host namenode1.gpfs.net, then run the following command:

     keytool -noprompt -importcert -alias namenode1.gpfs.net -file /etc/security/serverKeys/trust_store/namenode1.gpfs.net.pem -keystore /var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks

     This command prompts the user to enter the Cloudera Manager global Truststore password. To find the Truststore password, see /etc/hadoop/conf/ssl-client.xml on any Cloudera node.

   • Run the keytool -importcert command for all the .pem files, until all the certificates have been imported to Cloudera global Truststore.
   • Distribute the CES HDFS Truststore to all the HDFS Transparency nodes (NameNodes and DataNodes). For ease of use, you may use the following bash shell code snippet:

     cd /etc/security/serverKeys
     export HDFS_TRANSPARENCY_NODES="<space separated list of all HDFS Transparency hostnames FQDN"
     ## example: export HDFS_TRANSPARENCY_NODES=”nn01.gpfs.net dn01.gpfs.net dn03.gpfs.net"
     for hosts in $HDFS_TRANSPARENCY_NODES
     do
       scp spectrum_scale_ces_hdfs_truststore.jks ${hosts}:/etc/security/serverKeys
     done

   • Distribute all the modified IBM Storage Scale and Cloudera Truststore files and certificates (.pem files) to all the CDP Private Cloud Base cluster nodes, including all the IBM Storage Scale nodes registered to Cloudera Manager. For ease of use, you may use the following bash shell code snippet:

     export CMSTORES_DIR="/var/lib/cloudera-scm-agent/agent-cert/"
     cd /etc/security/serverKeys
     cp spectrum_scale_ces_hdfs_truststore.jks ${CMSTORES_DIR}/
     cd ${CMSTORES_DIR}/
     export ALL_NODES="<space separated list of all Cloudera hostnames FQDN"
     ## example: export ALL_NODES=”cldr1.gpfs.net cldr2.gpfs.net"
     for hosts in $ALL_NODES
     do 
       
       scp cm-auto-global_truststore.jks ${hosts}:${CMSTORES_DIR}
       scp cm-auto-global_cacerts.pem  ${hosts}:${CMSTORES_DIR}
     done
     

1. Go to Cloudera Manager.
2. Click IBM Spectrum Scale > Configuration > HDFS client Advanced Configuration Snippet (Safety Valve) for hdfs-site.xml and add the following custom configuration:

     <property>
       <name>dfs.namenode.https-address.<cluster name>.nn1</name>
       <value><CES_HOSTNAME>:50470</value>
     </property>
     <property>
       <name>dfs.http.policy</name>
       <value>HTTPS_ONLY</value>
     </property>
     <property>
       <name>dfs.client.https.need-auth</name>
       <value>false</value>
     </property>
   

   where, <CES_HOSTNAME> is the FQDN Hostname corresponding to the CES IP configured for your CES HDFS cluster.

   <cluster name> is the name of your CES HDFS cluster that is also your HDFS namespace.

   If you want both the secure and unsecure http connections, set dfs.http.policy to HTTP_AND_HTTPS.

3. Save and deploy the client configuration. Do not start any services yet.

1. Click IBM Spectrum Scale > Configuration and search for the following configurations and update them as follows:

   Spectrum Scale TLS Enabled = true
   HDFS Transparency DataNode HTTP Port = 1006
   HDFS Transparency NameNode HTTP Port = 50470

   Note: These configurations are needed for HDFS metrics to appear in Cloudera Manager when TLS is enabled.
2. Save and deploy the client configuration.

• If Ranger is enabled, start the NameNodes using the workaround mentioned in issue. Then start all the services from Cloudera Manager.
• If Ranger is not enabled, start all the services from Cloudera Manager as usual.