Resource usage limits for multidimensional scheduling

You can set limits on resource usage in multidimensional scheduling (for CPU or memory usage) for all service instances in an application, or for a single service instance on a host, based on multidimensional slot definitions.

Supported operating systems

Limiting CPU and memory usage in IBM® Spectrum Symphony is supported on , and Windows, Linux®, and Linux on POWER® systems. For Linux on POWER, only limiting memory (not CPU) is supported.

Setting limits for CPU or memory usage

CPU: An IBM Spectrum Symphony service consumes CPU time on a compute host. You can limit the number of cores on which an IBM Spectrum Symphony service is expected to run by configuring job objects for Windows (no configuration is required for job objects), and cgroups for Linuxs. You can then limit CPU usage, which is a hard limit. For example, if four cores are available on the host and the SI is configured for one, it will use only one core as a maximum.
Memory: You can limit the virtual memory usage for Windows, or the physical memory usage for Linux, for multidimensional scheduling. IBM Spectrum Symphony converts the maxmem multidimensional metric to either a virtual memory limit for Windows, or as a physical memory limit for Linux, in the IBM Spectrum Symphony application profile.
You can configure the maximum amount of memory that an IBM Spectrum Symphony service and its descendent processes can consume using job objects on Windows (no prerequisite configuration is needed for job objects, as the SIM creates a job object and once the SI process is created, the SIM attaches the SI to the job object and applies the limits), or using cgroups on Linux (first install a cgroup on each Linux compute host).

You can limit the CPU and memory for all service instances in an application, or for a single service instance on a host based on the multidimensional slot definitions for the ncpus and maxmem multidimensional metrics. IBM Spectrum Symphony converts these multidimensional metrics to CPU and memory fencing configuration. (Although IBM Spectrum Symphony also supports the maxswp and maxtmp metrics, only the ncpus and maxmem metrics are used to convert CPU and memory fencing configuration.)

Take, for example, this multidimensional scheduling plan, which specifies the ncpus and maxmem metrics:

<MDS planName="mdsPlan">
  <SlotDefinitions>
    <SlotDefinition name="mem">
      <Metric name="ncpus">1</Metric>
      <Metric name="maxmem">2000</Metric>
    </SlotDefinition>
  </SlotDefinitions>
</MDS>

With this configuration, IBM Spectrum Symphony converts the ncpus metric value of 1 to a cpuLimit parameter value of 1 in the IBM Spectrum Symphony application profile. Likewise, it converts the maxmem metric value of 2000 to a virtualMemoryLimit parameter value (for Windows), or to a physicalMemoryLimit parameter value (for Linux), of 2000 in the application profile.

Best practices and troubleshooting

Take note of the following scenarios when using resource usage limits for multidimensional scheduling:

On Windows, when you enable resource usage limits for multidimensional scheduling and if the memory used is more than the limitation set, the SIM receives a message from the Windows system to kill the SI. However, the Windows API does not always relay this message, and therefore, the SIM may not know to kill the SI.
To ensure that the SI will be restarted and the task will be retried as expected, as a best practice, in your service code, use a try-catch pair statement (try and catch) when applying memory to catch the out of memory message, and add an exit success statement (exit(0)) to indicate that the SIM has successfully killed the SI. For example:
```
try
{
    ... ...
    p[i]=new int[nSize];
    ... ...
}
catch(std::bad_alloc& ex)
{
    m_logger->log(MyLogger::INFO_LOG, "My alloc in onInvoke(), catch the bad_alloc and get memory failed");
    exit(0);
}
catch(...)
{
    m_logger->log(MyLogger::INFO_LOG, "My alloc in onInvoke(), catch the ... and get memory failed");
    exit(0);
}
```
On Windows and Linux, when you enable resource usage limits for multidimensional scheduling and if the memory used is more than the limitation set, the SIM kills the SI. This kill operation occurs during any SI phase, such as during the Register or CreateService methods. Next, the controlCode actions (such as blockHost), defined in the application profile, are triggered.
Therefore, if you find that there are sessions in open state and cannot be closed, or if you see blockHost messages in the SIM log, you may have blocked hosts. Use the Hosts Blocked for Application page within the cluster management console to see a list of blocked hosts, and then unblock them by clicking Unblock (see Removing a host from the blocked hosts list for details). You can also unblock hosts by re-registering the application profile using the soamreg command.