Platform Conductor for Platform LSF: Run LSF jobs using Platform Conductor resources

Body

If you’ve got IBM Platform LSF (LSF), set up your LSF cluster to borrow resources from an IBM Platform Conductor for Spark (Platform Conductor) cluster. With the resource connector set up, when the user submits jobs to LSF and the LSF cluster does not have resources to run the jobs, LSF borrows unused resources from the Platform Conductor cluster. When Platform Conductor reclaims the resources, LSF returns them – re-queuing jobs if necessary.

This resource sharing is achieved through EGO consumers and resource groups, where LSF and Platform Conductor consumers share, but the Platform Conductor consumer owns 100% of the resources. There's no new bsub command syntax that you have to learn to leverage this feature.

To set up the resource connector, ensure that you have LSF version 9.1.3 and Platform Conductor version 1.1. Then, download fix pack lsf-9.1.3-build372290 from IBM Fix Central, and install and enable the resource connector using the instructions in the readme.

Sample use case

Here's  a sample use case to illustrate resource sharing between LSF and Platform Conductor:

• Assume that we’d like to lend out the following hosts from a Platform Conductor cluster to an LSF cluster:

• Follow the readme to configure LSF for the resource connector.
• Follow the readme to configure Platform Conductor for resource sharing with LSF. In particular, create a new resource group containing the above hosts, and create an LSF consumer to share this resource group with a Platform Conductor consumer.

The following screenshot shows a sample resource plan where ‘rg_shared’ is the shared resource group; ‘mydemo-lsf’ is the consumer through which LSF allocates resources; and ‘mydemo-spark’ is the consumer through which Platform Conductor allocates resources:

• To trigger a borrow demand when there are no available resources in the LSF cluster, submit a job with no options. For example:
  • $bsub sleep 1000
• To directly specify a Platform Conductor host, submit a job with a resource requirement as follows (for example):
  • $ bsub -R "select[conductorhost]" sleep 1000
• Use bhosts to see which hosts from the Platform Conductor cluster joined the LSF cluster and what jobs are being executed on those hosts:

• If a user starts a Spark batch application for the Spark instance group in which the resource group is being shared, then LSF returns the borrowed hosts to Platform Conductor. Running jobs are re-queued.

We hope you can use the resource connector to take advantage of resource-sharing capabilities. If you’d like to try out Platform Conductor, download an evaluation version from the Service Management Connect page. If you have any questions, post them in our forum!