Securing Spark web interfaces

Complete this task to secure Spark web interfaces.

1. Generate a public-private key pair. Then, wrap the public key in a digital certificate, and store the private key and the certificate in a keystore. The following example uses the Java keytool tool to generate a self-signed certificate.

   keytool -genkeypair -keystore /u/sparkid/.keystore \
   -keyalg RSA -alias selfsigned \
   -dname "CN=mysparkcert L=Poughkeepsie S=NY C=US" \
   -storepass examplestorepass -keypass examplekeypass

2. Export the generated certificate and import it into a Java truststore. The following example again uses the Java keytool tool.

   keytool -exportcert -keystore /u/sparkid/.keystore \
   -alias selfsigned -storepass examplestorepass -file test1.cer
   
   keytool -importcert -keystore /u/sparkid/.truststore \
   -alias selfsigned \
   -storepass examplestorepass -file test1.cer -noprompt

3. Update the spark-defaults.conf file to enable SSL for Spark WebUI, by using the keystore and truststore that is setup in the previous steps.

   spark.ssl.enabled                   true
   spark.ssl.trustStore                /u/sparkid/.truststore
   spark.ssl.trustStorePassword        examplestorepass
   spark.ssl.keyStore                  /u/sparkid/.keystore
   spark.ssl.keyStorePassword          examplestorepass
   spark.ssl.keyPassword               examplekeypass
   spark.ssl.protocol                  TLSv1.2
   

4. Start your Spark cluster as normal. When you point your web browser to the Spark web interface, it automatically redirects to the SSL port, which is typically the non-SSL port plus 400. For example, http://127.0.0.1:8080 would be directed to https://127.0.0.1:8480.

   You can also use the spark.ssl.ui.port option to set the SSL port for the Spark web UI. The spark.ssl.ui.port option can be specified in spark-defaults.conf.

   Note: If you are using a self-signed certificate, like the one in the previous example, you might need to install the certificate in your web browser. Self-signed certificates are generally rejected by web browsers, since they are not signed by a known certificate authority and therefore not trusted.