Resource connector enhancements

The following enhancements affect LSF resource connector.

LSF resource connector auditing

With this release, LSF will log resource connector VM events along with usage information into a new file rc.audit.x (one log entry per line in JSON format). The purpose of the rc.audit.x log file is to provide evidence to support auditing and usage accounting as supplementary data to third party cloud provider logs. The information is readable by the end user as text and is hash protected for security.

New parameters have been added to LSF in the lsf.conf configuration file:

LSF_ RC_AUDIT_LOG: If set to Y, enables the resource connector auditor to generate log files.
RC_MAX_AUDIT_LOG_SIZE: An integer to determine the maximum size of the rc.audit.x log file, in MB.
RC_MAX_AUDIT_LOG_KEEP_TIME: An integer that specifies the amount of time that the resource connector audit logs are kept, in months.

Resource connector template prioritizing

In 10.1 Fix Pack 6, resource connector prioritizes templates.

The ability to set priorities is now provided in the resource connector template. LSF will use higher priority templates first (for example, less expensive templates should be assigned higher priorities).

LSF sorts candidate template hosts by template name. However, an administrator might want to sort them by priority, so LSF favors one template to the other. The “Priority” attribute has been added.:

{
      "Name": "T2",
      "MaxNumber": "2",
      "Attributes":
      {
        "type": ["String", "X86_64"],
        "ncpus": ["Numeric", "1"],
        "mem": ["Numeric", "512"],
        "template": ["String", "T2"],
        "ostkhost": ["Boolean", "1"]
      },
      "Image": "LSF10.1.0.3_OSTK_SLAVE_VM",
      "Flavor": "t2.nano",
      "UserData": "template=T2",
      "Priority": "10"
    }

Note: The example above is for a template in openStack. Other templates may not contain all attributes.

The default value of Priority is “0”, which means the lowest priority. If template hosts have the same priority, LSF sorts them by template name.

Support for a dedicated instance of AWS

One new parameter is added to the resource connector template to support a dedicated instance of AWS.

If you do not have a placement group in your AWS account, you must at least insert a placement group with a blank name inside quotation marks, because this is required to specify the tenancy. If you have a placement group, specify the placement group name inside the quotation marks. For example, "placementGroupName": "", or "placementGroupName": "hostgroupA",.

The values for tenancy can be "default", "dedicated", and "host". However, LSF currently only supports "default" and "dedicated".

The above can be applied for both on-demand and spot instances of AWS.

Full example the template file is as follows:

{
    "templates": [
         {
            "templateId": "aws-vm-0",
            "maxNumber": 5,
            "attributes": {
                "type": ["String", "X86_64"],
                "ncores": ["Numeric", "1"],
                "ncpus": ["Numeric", "1"],
                "mem": ["Numeric", "512"],
                "awshost": ["Boolean", "1"],
                "zone": ["String", "us_west_2d"]               
            },
            "imageId": "ami-0db70175",
            "subnetId": "subnet-cc0248ba",
            "vmType": "c4.xlarge",
            "keyName": "martin",
            "securityGroupIds": ["sg-b35182ca"],
            "instanceTags": "Name=aws-vm-0",
            "ebsOptimized" : false,
            "placementGroupName": "",
            "tenancy": "dedicated",
            "userData": "zone=us_west_2d"        }
}

HTTP proxy server capability for LSF resource connector

This feature is useful for customers with strict security requirements. It allows for the use of an HTTP proxy server for endpoint access.

Note: For this release, this feature is enabled only for AWS.

This feature introduces the parameter "scriptOption" for the provider. For example:

{
    "providers":[
        {
            "name": "aws1",
            "type": "awsProv",
            "confPath": "resource_connector/aws",
            "scriptPath": "resource_connector/aws",
           "scriptOption": "-Dhttps.proxyHost=10.115.206.146 -Dhttps.proxyPort=8888"
        }
    ]
}

The value of scriptOption can be any string and is not verified by LSF.

LSF sets the environment variable SCRIPT_OPTIONS when launching the scripts. For AWS plugins, the information is passed to java through syntax like the following:

java $SCRIPT_OPTIONS -Daws-home-dir=$homeDir -jar $homeDir/lib/AwsTool.jar --getAvailableMachines $homeDir $inJson

Create EBS-Optimized instances

Creating instances with EBS-Optimized enabled is introduced in this release to archive better performance in cloud storage.

The EBS-Optimized attribute has been added to the resource connector template. The AWS provider plugin passes the information to AWS when creating the instance. Only high-end instance types support this attribute. The resource connector provider plugin will not check if the instance type is supported.

The "ebsOptimized" field in the resource connector template is a boolean value (either true or false). The default value is false. Specify the appropriate vmType that supports ebs_optimized (consult AWS documentation).

{
    "templates": [
        {
            "templateId": "Template-VM-1",
            "maxNumber": 4,
            "attributes": {
                "type": ["String", "X86_64"],
                "ncores": ["Numeric", "1"],
                "ncpus": ["Numeric", "1"],
                "mem": ["Numeric", "1024"],
                "awshost1": ["Boolean", "1"]
            },
            "imageId": "ami-40a8cb20",
           "vmType": "m4.large",
            "subnetId": "subnet-cc0248ba",
            "keyName": "martin",
            "securityGroupIds": ["sg-b35182ca"],
            "instanceTags" : "group=project1",
            "ebsOptimized" : true,
            "userData": "zone=us_west_2a"
        }
    ]
}

Resource connector policy enhancement

Enhancements have been made for administration of resource connector policies:

A clusterwide parameter RC_MAX_REQUESTS has been introduced in the lsb.params file to control the maximum number of new instances that can be required or requested.
After adding allocated usable hosts in previous sessions, LSF generates total demand requirement. An internal policy entry is created as below:
```
{
      "Name": "__RC_MAX_REQUESTS",
      "Consumer": 
       {
        "rcAccount": ["all"],
        "templateName": ["all"],
        "provider": ["all"] 
       },      
      "StepValue": "$val:0"   
    }
```
The parameter LSB_RC_UPDATE_INTERVAL controls how frequent LSF starts demand evaluation. Combining with the new parameter, it plays a cluster wide “step” to control the speed of cluster grow.