Setting Docker/cgroup container definitions for an instance group

Optionally set up the instance group to run within containers, either Docker or control groups (cgroups).

1. In the Containers tab, specify the container settings for Spark drivers, Spark executors, or Spark services in the instance group. By default, Spark drivers, Spark executors, and Spark services run without containers.
   • To run Spark drivers, executors, or services in Docker containers, click Docker containers.
   • To run Spark drivers and executors in cgroup containers, click cgroups.
2. If you chose to run Spark drivers, executors, or services in Docker containers, create a definition for the Docker container.
   1. Click New definition.
   2. Enter the Definition name and the Docker image name.
   3. Optional: If the Docker image that you specified is not loaded on the Docker installation, enter a Docker registry URL. The URL of the public Docker registry (https://hub.docker.com/) is used by default.
      Note: If your host is not connected to the Internet, specify either the URL to an internal Docker registry that all hosts can access or load the image locally on all hosts.
   4. Optional: Add one or more host-level data volumes, which are mount points for the Docker container.
      1. Click Add a data volume.
      2. Enter the host path to the directory in the host OS for use inside the container, for example: /myHostpath/path.
      3. Enter the container path to the directory that is to be mounted inside the container, for example: /container/myHostpath/path.
      4. Enter the environment variable that must be exported from the host to the container. For example, you could use the DIR_ENV_VAR variable to represent a directory path, such as /container/myHostpath/path, enabling access to the mounted directory.
      5. Click Writable to specify read/write (rw) access to the directory inside the container.
      6. Repeat the steps to mount all the directories that your applications require to run in the container.
      Note: If VEMKD is enabled, the file path of the SSL authentication certificate must be mounted to Docker, and the instance group execution user must have access to that directory.
   5. Click OK.

      The definition that you created is used as the default definition for Docker containers.

   6. Repeat the steps to create as many definitions as you require. With multiple Docker definitions, you can override the default container definition that Spark drivers or executors use when you submit applications.
3. Optional: Change the default per-resource limits for the container (both Docker and cgroup). You can change the default Spark version configuration to set memory limits for drivers and executors (by default, 1 GB). CPU shares are dynamically adjusted according to the number of slots that are allocated to the executor.
   1. Click the Basic Settings tab in the wizard.
   2. Click the Configuration link for the Spark version.
   3. To set memory limits for drivers and executors in the container, set the spark.driver.memory and spark.executor.memory parameters. Both parameters by default use 1 GB.
   4. Click Save.