Enabling Hadoop encryption

You must enable encryption in Hadoop.

Procedure

  1. Create a service user. For example, useradd kms.
  2. Copy the Hadoop-KMS package to the home directory.
    For example, enter the following command:
    cp /usr/iop/current/hadoop-client/mapreduce.tar.gz /home/kms/mapreduce.tar.gz
  3. Extract the archive.
    For example, enter the following command:
    export KMS_ROOT=/home/kms/ 
cd $KMS_ROOT 
tar -xvf mapreduce.tar.gz
  4. Start the KMS server.
    1. If you do not have the JAVA_HOME variable set, run the following command: 
      export JAVA_HOME=/usr/jdk64/java-1.8.0-openjdk-1.8.0.77-0.b03.el7_2.x86_64/jre

      Ensure that you use the appropriate path for your environment.

    2. Go to the $KMS_ROOT/hadoop/sbin/ directory.
    3. Enter the following command: ./kms.sh run

      Wait until you see that the server started.

  5. From the Ambari console, update the KMS server.
    1. In the Ambari console, click the HDFS service.
    2. Click Configs > Advanced.
    3. Add the following values:
      Configuration section Key Value1
      Advanced core-site hadoop.security.key.provider.path kms://http@<KMS Server IP>:16000/kms
      Advanced hdfs-site dfs.encryption.key.provider.uri kms://http@<KMS Server IP>:16000/kms
  6. Generate a key as a regular user.
    1. Log on as a regular user, such as ambari-qa.
    2. Create the key by entering the following command: hadoop key create ambariqa-key
  7. Create an encryption zone for the /user/sifsuser directory.
    1. Log in as the hdfs user.
    2. Run the following commands: 
      hdfs crypto -createZone -keyName ambariqa-key -path /user/sifsuser
hdfs dfs -chown sifsuser:hadoop /user/sifsuser
      Tip: If you encounter any errors, you can check the following log directories:
      • /var/log/hadoop/hdfs
      • /var/log/ambari-server
      • /var/log/ambari-agent
      • /var/lib/ambari-agent/data
  8. Verify that the contents are encrypted.
    1. Log in as the sifsuser.
    2. Copy a test data file to the /user/sifsuser directory.
    3. Run the following commands: 
      hdfs dfs –put testdata.txt /user/sifsuser/
hdfs dfs -cat /user/sifsuser/testdata.txt   
hdfs dfs -cat /.reserved/raw/user/sifsuser/testdata.txt

      This should show decrypted, clear text data.

      Run the following command:

      hdfs dfs -cat /.reserved/raw/user/sifsuser/testdata.txt

      This should show encrypted data.

      Note: If the Kerberos session has expired, you can run the kinit command.