Creating instance groups

Each instance group is a Spark installation that can run Spark core services (Spark master, shuffle, and history) and notebooks as configured. You can create multiple instance groups, each using a different Spark version, and associate each instance group with packages that its applications require to run.

Instance groups that require resources do not communicate directly with the resource orchestrator. Instead, they request resources through the consumer that they are associated with. In turn, the consumer is associated with one or more resource groups, whose hosts provide resources for the instance group. Each instance group includes core services, such as Spark drivers, executors, the shuffle, and batch master. Depending on your configuration, it might also include notebook and history services.

When you create an instance group, your cluster configuration and permissions determine the settings that are available to you. As a result, you may not see all available options.