Shuffle data security through SASL

You can enable shuffle data security by using simple authentication and security layer (SASL) for the instance group and Spark applications.

Before you begin

Restriction: If security settings are enforced at the cluster level, you cannot change these settings for the instance group. Talk to your cluster administrator for more information.
  • Spark versions not supported: 1.5.2, 2.0.1, and 2.1.0.
  • Based on your requirements, ensure that you meet the requirements to create an instance group. See Prerequisites for an instance group.

About this task

You can enable SASL for a Spark application within an instance group to ensure that only executors started by the application get shuffle data. In the Spark configuration, you can set the open source Spark parameter spark.authenticate to true, which turns on instance group security protection and makes shuffle service data secure. Do not configure the other open source Spark parameters spark.authenticate.enableSaslEncryption and spark.network.sasl.serverAlwaysEncrypt.

Procedure

  1. From the cluster management console, click Workload > Instance Groups.
  2. In the Basic Settings, select the Spark version that the instance group must use.
  3. Click Configuration to customize the settings for the Spark version.
    1. In the Security section, set spark.authenticate to true.

What to do next

  1. Finish configuring the instance group. See Defining basic settings for an instance group.
  2. Create and deploy the instance group.
    • Click Create and Deploy Instance Group to create the instance group and deploy its packages simultaneously. In this case, the new instance group appears on the Instance Groups page in the Ready state. Verify your deployment and then start the instance group.
    • Click Create Only to create the instance group but manually deploy its packages later. In this case, the new instance group appears on the Instance Groups page in the Registered state. When you are ready to deploy packages, deploy the instance group and verify the deployment. Then, start the instance group.