Creating an instance group for an IBM Spectrum Conductor cluster running within an IBM Spectrum LSF cluster

Once you have an IBM® Spectrum Conductor cluster within an IBM Spectrum LSF® (LSF) cluster, you can create instance groups and submit Spark applications.

Once you have started an instance group you can submit Spark applications that run as LSF jobs. By default, the Spark applications use default LSF parameters that are set for the LSF cluster. If you want to use LSF parameters which are different from the default LSF parameters, you can configure LSF parameters for an instance group. See Setting IBM Spectrum LSF parameters for an instance group.

Each Spark application runs as one or more LSF jobs, under the execution user specified for the application.

An LSF job group is created for each Spark application to group the LSF jobs associated with the Spark application. The names of the created job groups are hierarchical, and include the following components: Conductor cluster name, instance group name, execution user name, and application ID. These job groups can be removed when no longer required.

An LSF job group is created for the IBM Spectrum Conductor management cluster. The name of this job group starts with Conductor_management. This LSF job group should not be used for jobs which are not generated by the IBM Spectrum Conductor cluster. This LSF job group must not be removed as long as IBM Spectrum Conductor is installed. This LSF job group can be removed when IBM Spectrum Conductor is uninstalled.

The IBM Spectrum Conductor cluster creates and uses LSF advance reservations to allocate resource slots on hosts in the LSF cluster for IBM Spectrum Conductor workload. The IBM Spectrum Conductor workload is launched as LSF jobs that reference these advance reservations. The advance reservations are increased and decreased with resource slots dynamically according to workload demand, and are automatically removed when no longer required.

Note: The LSF advance reservations created and used by IBM Spectrum Conductor must not be removed. Removing these advance reservations will cause applications and services to fail. In a situation of erroneous removal of these advance reservations, stop and restart the affected instance groups. The LSF advance reservations created and used by IBM Spectrum Conductor are named using the following name format:

cluster_name:
maximum_10_characters_instance_group_name:
maximum_10_characters_type:
2_characters_unique_ID

With the LSF integration, the option to configure consumers or resource groups in IBM Spectrum Conductor is not applicable.