Data-aware scheduling

Plug-in support

The IBM Spectrum Scale-FPO data-aware scheduling plug-in is supported on Linux®.

About data-aware scheduling

Workload schedulers focus on dispatching tasks to compute hosts and transferring data either directly to the compute hosts or delegating data retrieval to the service. The time required for this data transfer from various sources to where the work is being processed can lead to inefficient use of CPU cycles and under-utilization of the resource. With the data-aware scheduling feature, you can specify a preferential association between a task and a service instance or host that already possesses the data required to process the workload. This association is based on the evaluation of a user-defined expression containing data attributes capable of being collected. The evaluation of this expression is carried out against the data attributes of each service instance available to the session. Typically, a data attribute is an identifier for a data set that is already available to a service instance before it processes workload.

The following example illustrates the concept of data-aware scheduling at the task level. The data preference expression has been evaluated and it is determined that a task in the queue prefers to run on a service instance where the service instance already possesses Dataset1. The SSM collects meta data (service attributes) from all the resources available to the session at that moment. Service B with Dataset1 is currently available and since it is the best match for that task according to the specified preference, the task is subsequently dispatched to Service B.

Expressing a preference for data

Specifying data attributes in a preference expression

Data attributes take the form of identifiers and can be specified with the +, -, *, and / operators within an expression. Each data attribute resolves to a number. The operators are math operators (+ means add, - means subtract, * means multiply, and / means divide). An example of an expression would be DataSet1 + DataSet2, meaning that it is preferred to send workload to a service instance that possesses DataSet1 plus DataSet2. The operators can also be used to normalize disparate terms such as data and memory or to give weight to specific terms.

For example, in the flow of task-level data aware scheduling, each task can have a preference expression, comprised of resource attributes and operators. Each host or service instance can have a different value for each resource attribute. This way, each task's preference expression can be evaluated against each host. The host or service instance that yields the lowest value preference expression is considered the best host or service instance for that task.

Attribute names have a 32 character limit and can only contain alphanumeric and underscore characters; if you want to use data attributes with names that do not comply with these rules, define aliases. A resource attribute definition can be used to define an alias and to override the default value for an attribute should the session level default or system default be inappropriate.

How IBM Spectrum Symphony handles the result of the expression

The result of each expression is a numeric value that is obtained by applying the operators to the attributes in the expression. If the preferred data is available to a service, it should programmatically publish a value for the attribute. Alternatively, the value of the attribute may be collected from the data-aware scheduling plug-in, if present. Once the result is obtained for each resource being evaluated, it is used to sort the resources in ascending order. This means the resource that evaluates to the lowest value is the most preferred.

When no information is available for a resource attribute that is involved in a resource evaluation, the resolution of the expression still proceeds. In such cases, Symphony attempts to substitute a default value for each attribute that it cannot resolve. The value of the attribute is resolved by the system in the following manner, in the given order:

How preference affects dispatch order

Once a preference is specified on a task within a session, the best match for that task with every service instance currently assigned to the session is considered during the dispatching of the task. This means that once a service becomes available, it is given the next task with the best match for that service instance at the current moment. It is important to note that the next task to be dispatched may not currently be at the front of the queue within the session, i.e., the order of task dispatch depends on the currently available service instances and the preference associated with the tasks and not on the order that tasks are submitted to the session.

Note that the data-aware scheduling feature does not affect the behavior of tasks with the PriorityTask setting enabled. Those tasks are still dispatched before other tasks.

Factoring the cost of data transfer

When you want to specify a preference for specific data, the expectation is that attributes in the expression would have their default value set to some number representing the cost of moving this data to that resource.

For example, consider the case where a task within a session requires data sets Dataset1, Dataset2 and Dataset3, the preference for the task could be represented as Dataset1 + Dataset2 + Dataset3. Since expressions are evaluated and sorted in ascending order, you would expect the most preferred resource for this task would have the lowest cost to get the data that the task requires to execute.

Since a service instance is typically able to inform the middleware about what resource attributes (that is, data sets) it has access to, a fixed value can be substituted for any missing attributes when evaluating the task preference against this service instance. If you use the same fixed value to represent each missing attribute, the system will assign each missing data set the same cost so service instances that have access to the most data sets would be the most preferred and so on. But in reality, it is not generally true that all missing data sets would have the same cost to acquire, especially in the case where the data sets can vary greatly in size or be located on hosts with disparate network topologies and network access speeds. That is why to gain the most benefit from the data-aware scheduling feature, the default value for a data attribute should be carefully considered when applied to a preference expression to ensure that it is representative in some way of the cost of retrieving the missing data for the service instance. This default value is set through the resource attribute definition API.

Data-aware scheduling plug-in

The plug-in process requests the information from an external application and returns it back to the SSM to be used by IBM Spectrum Symphony as the preferred scheduling location and value for the attribute in the same way as if it were published by a service. Sample code for the plug-in process is provided in an appendix at the end of this document.

To improve performance, this feature also provides caching for the result received from the plug-in so subsequent calls for the same attribute will return cached values.

Here is the sequence the SSM follows to obtain the resource attribute value when data-aware scheduling is enabled:

If the plug-in is configured and the preferred attributes do not have any published values during the current evaluation cycle, the SSM requests locations and values from the plug-in. The following describes the functional flow when the plug-in is used to retrieve resource attribute data for a task.

1. A task is submitted with one or more preference attributes.
2. Upon the task's submission, the SSM parses the preferences and puts the task in the pending queue.
3. The preferences are evaluated and the SSM tries to find values for the corresponding attributes. Since, in this case, the attribute was not published by the service, the expression is not resolved.
4. The SSM checks if the direct plug-in is configured for the application. The direct plug-in process is started using a startup command defined in the application profile.
5. The SSM repeatedly calls the plug-in interface with the attribute names to retrieve the corresponding locations and values.
6. Tasks are dispatched to currently available resources for the session that are the best match according to the preferred attributes.

IBM Spectrum Scale-FPO data-aware plug-in

When using the IBM Spectrum Scale-FPO data-aware plug-in, also configure host mapping when multi-homed hosts are configured on the hosts. If multi-homed hosts are configured on the host, to ensure that the IBM Spectrum Scale-FPO host name and IBM Spectrum Symphony host names are the same, configure the EGO host file ($EGO_CONFDIR/hosts (Linux) or %EGO_CONFDIR%\hosts (Windows)) with the correct mapping information.

Configuring data-aware scheduling

Enabling data-aware scheduling for tasks

Data-aware scheduling is enabled at the application level with the schedulingAffinity attribute in the Consumer element of the application profile. When the attribute is set to DataAware, the SSM collects data attributes of service instances and hosts and evaluates them against a user-defined preference expression. Note that setting the attribute to DataAware automatically enables resource-aware scheduling. When the attribute is set to None (default), no metadata is collected by the SSM and no preference is applied.

<Consumer applicationName="SharingDataCPP" ...schedulingAffinity="DataAware" />

The schedulingAffinity attribute can be configured through the cluster management console or by manually editing the application profile.

Configuring the data-aware plug-ins

<SSM resReq="" workDir="${EGO_SHARED_TOP}/soam/work"
 startUpTimeout="60" shutDownTimeout="300" SSMResPubPluginCmd="C:\extplugin.exe"
 SSMResPubPluginCmdTimeout="60" SSMResPubCacheEnabled="false"> 
</SSM>

<SSM SSMResPubPluginCmd="<plugin_path 
GPFS_FPO_MMLSNSD GPFS_FPO_MMLSDISK 
[-d DIRECTORIES] [-p POLLING_INTERVAL] [-c CACHE_ENABLED] [–e CACHE_ENTRIES]> 
"enableOptimizedHostQueue="true"/>

<SSM SSMResPubPluginCmd="</…/gpfs_fpo_das_plugin 
/usr/lpp/mmfs/bin/mmlsnsd /usr/lpp/mmfs/bin/mmlsdisk 
-d /gpfs/f1 -p 120 -c true -e 10000>" 
enableOptimizedHostQueue="true"/>

export LD_LIBRARY_PATH= $HADOOP_HOME/lib/native:$LD_LIBRARY_PATH

Developing clients and services for data-aware applications