Pre-provisioning and post-provisioning

Set up pre-provisioning in LSF resource connector to run commands before the resource instance joins the cluster. Configure post-provisioning scripts to run clean up commands after the instance is terminated, but before the host is removed from the cluster.

The pre- and post-provisioning script knows the instance ID, the instance name, and the instance IP address.

The pre-provisioning script runs on the management host right after the resource instance is created, but before the instance is marked as allocated to the LSF cluster, and before a job can start to run on it. Use the pre-provisioning script to run instance setup scripts (for example, network or user access configuration), run data transfer commands before the job starts on the instance, or add hosts to a specific group.

The post-provisioning script runs after the instance is terminated, but before it is relinquished to the provider. Use the post-provisioning script to run clean up tasks; for example, clean up job files or remove the host from the host cache file when the instance terminates.

You can specify a delayOnReturn attribute in your scripts that specifies the number of minutes that the resource connector waits before it returns the host in case the pre- or post-provisioning script fails. The default value is 20. For the scripts that are run successfully, resource connector does not apply the delay, even if delayOnReturn is set.

Enable pre- and post-provisioning

To enable pre- and post-provisioning scripts, set the following parameters in the hostProviders.json file:

preProvPath

Resource connector runs the pre-provisioning script that is specified with absolute path after the instance is created and started successfully but before it is marked allocated to the LSF cluster.

postProvPath

Resource connector runs the post-provisioning script that is specified with absolute path after the instance is terminated successfully but before it is removed from the LSF cluster.

provTimeOut

This parameter is used to avoid the pre- or post-provisioning program from running for unlimited time. Specify a value in minutes.

If the program doesn’t complete in the specified time, it is ended and reported as failed. You can disable pre- or post-provisioning by setting the provTimeOut value to 0.

The default value is 10 minutes. If the pre- or post-provisioning program doesn't return after 10 minutes, it ends.

Pre-provisioning and post-provisioning script output

Pre and post provision scripts are expected to return results to inform LSF is the hosts have been provisioned correctly or not. The output needs to look like the following example:

[{
"machines": [
          {
            "name": "instance name",
            "result": "succeed",
            "message": "User added successfully"
           },
          {
            "name": "ip-192-168-0-154.us-west-2.compute.internal",
            "result": "failed",
            "message": "Could not resolve host name”,
            “delayOnReturn” : 30
          }
    ]
}]

Example input.json file

The input.json file is the data in json format which is sent by LSF to the user’s provision scripts as input. The user defined provisioning scripts can use the input data of the AWS instances launched by LSF to do the necessary provisioning steps. The input.json file contains the following information:

[ {
"machines": [
          {
            "name": "ip-192-168-0-153.us-west-2.compute.internal",
            " publicIpAddress ": "55.66.xx.xx",
            " privateIpAddress ": "55.66.xx.xx",
            " machineId ": "i-19034xxxxx",
            “rcAccount”: “project1” ,
            "providerName": "aws",
            "providerType": "awsProv",
            "templateName": "Template-VM-2"
          }
    ]
}]

Example

This example uses the jq package to parse the input.json from LSF for each of the automatically launched AWS instances. After parsing the instance details the user can customize code to configure any DNS settings or cleanup steps for each instance before passing a success or error result back to LSF in the specified format.

LSF reads the output to determine if the newly launched host was successfully provisioned.

#!/bin/sh
inputFile=$1
outputFile=$2
echo $inputFile $outputFile

count=$(cat $inputFile | jq '.machines[]' |grep 'name' | wc -l)

a=0
result="succeed"

while [ $a -lt $count ]
do
  hostName=$(cat $inputFile | jq '.machines['${a}'].name')
  privateIp=$(cat $inputFile | jq '.machines['${a}'].publicIpAddress')
  publicIp=$(cat $inputFile | jq '.machines['${a}'].privateIpAddress')
  instanceId=$(cat $inputFile | jq '.machines['${a}'].machineId')
  rcAccount=$(cat $inputFile | jq '.machines['${a}'].rcAccount')
  providerName=$(cat $inputFile | jq '.machines['${a}'].providerName')
  providerType=$(cat $inputFile | jq '.machines['${a}'].providerType')
  templateName=$(cat $inputFile | jq '.machines['${a}'].templateName')

  #add your custom code here for each machine in the request
  #write the output of each machine to the output json file
  
  sed -i '/]/i {\"name\": '${hostName}', \"result\": \"'${result}'\", \"message\": \"'${message}'\" }' $outputFile
  a=`expr $a + 1`
done