Configuring IBM Bluemix for LSF resource connector

Follow these steps to configure the IBM Bluemix (formerly SoftLayer) service to enable LSF resource connector to make allocation requests to borrow virtual compute hosts on behalf of LSF. When LSF workload demand exceeds cluster capacity, resource connector generates requests for additional hosts from IBM Bluemix and dispatches jobs to dynamic hosts that join the LSF cluster. When the demand reduces, and hosts become idle, resource connector shuts down server LSF daemons and cancels allocated IBM Bluemix virtual servers.

1. Manually order a SoftLayer computing instance with a standard OS image.
   1. Log in to your account from the SoftLayer Customer Portal (https://control.softlayer.com/).
   2. Click Devices in the Order section and order a virtual server. Select the desired configuration for this virtual server.
   3. In Advanced System Configuration, choose a Host and Domain Name.
   4. Submit the order.

      You receive an email notification indicating that the virtual server is currently in the provisioning process. Standard delivery times are 20 minutes or less.

   5. Use Check Devices > Device List in the SoftLayer portal to see whether the cloud server is provisioned.

      After the server is available, you can customize this compute instance.

2. Customize the compute instance by installing IBM Spectrum LSF on it.

   To customize the virtual server instance, use SSH (with the public IP as root user). Obtain the root password for the device from the Bluemix portal by clicking the device name and selecting the Passwords tab in the device details.

   1. Use the scp command to copy the IBM Spectrum LSF installation packages to the Softlayer instance.

      lsf10.1_lnx310-lib217-x86_64.tar.Z
      lsf10.1_lsfinstall_linux_x86_64.tar.Z
      lsf_std_entitlement.dat

      Use the binary package appropriate for your virtual server OS version.
   2. Log on as root.

      ssh 169.55.162.62 -l root

   3. Configure DNS.

      Configure the DNS settings to make sure that the virtual server can look up the host name and IP address of the LSF management host.

   4. Edit the server.config file to configure the LSF installation.

      Extract the lsf10.1_lsfinstall_linux_x86_64.tar.Z file into a working directory.

      Typical contents of the server.config files are shown below (the values shown are examples. Use values appropriate for your environment):

      LSF_TOP="/home/bmuser/lsf"
      LSF_ADMINS="bmuser"
      LSF_TARDIR="/home/bmuser/lsf/"
      LSF_ENTITLEMENT_FILE="/home/bmuser/lsf/lsf_std_entitlement.dat"
      LSF_SERVER_HOSTS="management.myserver.com"
      LSF_LOCAL_RESOURCES="[resource softlayercomp] [resource define_ncpus_threads]"
      LSF_LIM_PORT="7869"

   5. Install the ed application, which is required.

      Use yum install command to install ed.

      yum install ed

   6. Install LSF on the server host.

      ./lsfinstall -s -f server.config

      For full details on installing LSF, see Installing IBM Spectrum LSF on UNIX and Linux.

   7. Start the LSF daemons on the virtual server manually and make sure that it can join the LSF cluster of the configured management host as a dynamic host.

      lsf_daemons start

   8. Stop the LSF daemons on the virtual server.

      lsf_daemons stop

3. Create the custom image template.

   By saving the previously customized virtual server image as a template you are able to provision a preconfigured LSF server host at any time.

   Go to Device Details for the host at Devices > Device List > Device Details > Actions, choose an Image Name (for example, LSFComputeImage), and click Create Image Template. ​

   When image creation is successful, go to Devices > Manage > Images to view the image you created. Remember the Image Name for resource connector configuration.

4. Create a provisioning script for the LSF compute node virtual server.

   IBM Bluemix can download the provisioning script from an HTTPS server and run it automatically when the virtual server is started and all its virtual devices are connected. The provisioning script is required to configure the LSF compute node to join the correct cluster and present correct shared resources. You can host your provisioning script on another virtual server in IBM Bluemix and configure it to use only the IBM Bluemix internal network for increased security. You can also host your script on any HTTPS server available on the public network.

   An example provisioning script is shown below. It reads from the getUserMetadata API to get the configuration variables set by the resource connector during provisioning.

         #!/bin/bash
         logfile=/var/log/postprovisionscripts.log
         echo START `date '+%Y-%m-%d %H:%M:%S'` >> $logfile
         
         #Do not remove this part of the script to support passing LSF user data to VM run time environment
         STARTTIME=`date +%s`
         TIMEOUT=60
         URL="https://api.service.softlayer.com/rest/v3/SoftLayer_Resource_Metadata/getUserMetadata.txt"
         USERDATA=`curl -s $URL` 2>>$logfile
         #
         while [[ "$USERDATA" == [Nn]"o user data"* ]] && [[ `expr $NOWTIME - $STARTTIME` -lt $TIMEOUT ]]; do
            sleep 5
            NOWTIME=`date +%s`
            USERDATA=`curl -s $URL` 2>>$logfile
         done
         
         # check if we got user data eventually
         if [[ "$USERDATA" != [Nn]"o user data"* ]]; then
            # user data is expected to be a semicolon-separated key=value list
            # like environment variables; split them into an array
            IFS=\; read -ra ARR <<<"$USERDATA"
            for VAR in ${ARR[@]}; do
               eval "export $VAR"
            done
         else
            echo "USERDATA: $USERDATA" >>$logfile
            echo EXIT AT `date '+%Y-%m-%d %H:%M:%S'` >>$logfile
            exit -1
         fi
         echo "CURRENT ENVIRONMENT:" >>$logfile
         env >> $logfile
         
         #Set the correct path for LSF_TOP, where LSF is installed on the VM host
         LSF_TOP=/opt/lsf
         LSF_CONF_FILE=$LSF_TOP/conf/lsf.conf
         source $LSF_TOP/conf/profile.lsf
         
         #Do not remove this part of the script to support rc_account resource for SoftLayer
         #You can similarly set additional local resources if needed
         if [ -n "${rc_account}" ]; then
            sed -i "s/\(LSF_LOCAL_RESOURCES=.*\)\"/\1 [resourcemap ${rc_account}*rc_account]\"/" $LSF_CONF_FILE
            echo "update LSF_LOCAL_RESOURCES lsf.conf successfully, add [resourcemap ${rc_account}*rc_account]" >> $logfile
         fi
         
         #If there is no DNS server to resolve host names and IPs between management host and VMs, 
         #then uncomment the following part and set the correct management LSF host name and IP address 
         #management_host='hostm.example.com'
         #management_host_ip='10.115.206.151'
         #echo ${management_host_ip} ${management_host} >> /etc/hosts
         #echo $management_host > $LSF_ENVDIR/hostregsetup
         #lsreghost -s $LSF_ENVDIR/hostregsetup
         
         #Start LSF Daemons in dynamic VM host.
         lsf_daemons start
         
         echo END AT `date '+%Y-%m-%d %H:%M:%S'` >> $logfile