Hybrid scheduling policy

The hybrid scheduling policy combines elements of both the ownership and sharing policies. It is applicable to all operating systems supported by IBM® Spectrum Symphony.

The key features of the hybrid policy are as follows:
  1. Resources can be owned and reserved. Reserved resources are excluded from lending and are always dedicated to the exclusive use of one consumer.
  2. Sharing (borrowing and lending resources) is enabled among all consumers. Labor-intensive maintenance of configuring peer-to-peer relationships is not required.
  3. When consumers need to borrow more resources than they own, the borrowed resources are distributed in proportion to the amount of resources each consumer owns.

Comparing policies

The following table summarizes the attributes of each scheduling policy:
Table 1. Scheduling policy attributes
Feature Hybrid Ownership Share ratio
Sharing by default Yes No Yes
Reserve slots from being shared Yes Yes No
Plan configured by absolute number Yes Yes No
Sibling first borrowing Yes No Yes
Balance checking Yes Yes No
Proportional borrowing Yes No Yes
Proportional reclaim Yes No Yes
Proportional ownership Yes No No

Basic concepts of the hybrid policy

Basic policy configuration for each consumer that can use the resource group includes the following:
  • owned
  • reserved
  • limit
Basic policy configuration for each consumer that can use the resource group
Owned resources

Default = 0. By default, a consumer only borrows slots, it does not own any.

A consumer that owns slots can use them whenever it needs them, even if it has to reclaim them from another consumer. Owning slots in the hybrid policy guarantees that a consumer is entitled to a specific number of slots at any time, even if other consumers also need slots.

Reserved resources

Default = 0. By default a consumer makes all its owned slots available for lending, and does not reserve any. Reservation is only possible if the consumer owns the slots.

A consumer's owned slots are either reserved or available for lending to other consumers (when they are not needed). A consumer can reserve some, all, or none of its owned slots.
  • Reserved slots cannot be lent to other consumers. When reserved slots are not needed by the owners, the system reserves enough idle slots to satisfy the potential demand. When an owner needs a reserved slot, it is available immediately.
  • If an owned slot is not reserved, it can be used by other consumers and then reclaimed when it is needed.
Note: Reserving slots can result in the best performance for a consumer, because tasks start without any delay. However, it usually results in the worst efficiency overall, because slots reserved by one consumer can be idle while another consumer has pending tasks.
Resource limit

Default = unlimited. By default, there is no fixed limit to the number of slots a consumer can use.

If you want to prevent a consumer from using too many slots at one time, set a limit. The limit is compared to the total number of slots used by a consumer, it does not matter if the slots are owned, or borrowed, or a combination of both.

Note: Limiting a consumer's resource usage can actually improve the efficiency across multiple consumers, in cases where consumer workload changes quickly and it would take a long time to redistribute those resources to other consumers. Resource limits prevent the system from taking demand-based slot allocation to extremes. Resource limits help you enforce a more balanced distribution of slots among consumers.

Advanced concepts of the hybrid policy

If you want more control over the way slots are shared among consumers, familiarize yourself with the following concepts:
  • hierarchy
  • rank
  • quota
  • slot distribution
  • reclaim
  • free slots

Consumer hierarchy

To distribute slots to leaf consumers, the system does not necessarily compare all the leaf consumers at once. Instead, the system considers how to divide all the available slots among the root-level consumers. If any consumers have sub-consumers, the slots assigned to each parent are then shared among its child consumers.

Comparing consumers in a hierarchy

Comparing consumers in a hierarchy

Child consumers with the same configuration and the same parent should get the same number of slots. However, child consumers with the same configuration but different parents may get a different number of slots.

In this example, all consumers are busy, and child consumers A,B,C, and D are all configured the same, but parent consumers X and Y are not. The resource group has 60 slots, but X is assigned 10 and Y is assigned 50. This imbalance affects the child consumers: even though the leaf consumers all have equal demand, A and B must share the 10 slots from X (5 each) while C and D share 50 slots from Y (25 each).

Consumer rank

Sometimes the system must choose to distribute a slot to a single consumer, but there are multiple consumers that are equally entitled it. This is when consumer ranking matters.

If you configure consumer rank, the system distributes slots to the highest-ranked consumers first. Among consumers of equal rank, the system follows the order that they appear in the resource group's resource plan.

If the EGO_CONSUMER_PRIORITY_DEFAULT parameter is not defined in ego.conf, all consumers have the highest possible rank (0). You can lessen a consumer's rank by specifying a higher numeric value.

Comparing consumer rank

If child consumers have the same parent, it is useful to compare their rank. However, it is not useful to compare the rank of child consumers that have different parents.

In this example, all leaf consumers have equal demand. The parent consumers X,Y, and Z all own the same number of slots, so they are equally entitled to additional resources. However, they have different ranks. If there is only one slot available, it goes to the highest-ranking consumer at the root level (Y) and then passes to its highest-ranking child (D).

The high ranks of consumers E and F do not affect this decision, because their parent consumer had a lesser ranking.

Quota calculations

Every scheduling cycle, the system re-evaluates the number of slots from each resource group that each consumer is entitled to use at that time. We call this dynamic number the consumer's "quota".

Even if configuration stays the same, the quota may change because of changes in workload or slot availability. After calculating the new quotas, the system reallocates slots, so each consumer gets the appropriate amount.

The system tries to set every consumer's quota so it has enough slots to run all its tasks. In a resource group that has plenty of available slots, this may be possible. When there is more competition for resources, the way you configure the policy determines which consumers are the most affected by an overall shortage of resources.

Assuming no reserve or limit, the quota is based on 3 calculations:
  1. How many slots does the consumer need?
  2. How many slots does the consumer own?
  3. How many slots can the consumer borrow?

How many slots does the consumer need?

The number of slots a consumer needs from the resource group is called the "demand".
  • For a leaf consumer, it corresponds to the number of slots required by applications.
  • For a parent consumer, the demand is calculated by adding up the demand of each child consumer.

To prevent waste, the quota never exceeds the demand (assuming the consumer has no reserved slots).

How many slots does the consumer own?

The number of slots a consumer owns is proportionally adjusted, and depends on the number of slots in the resource group. The number of slots a consumer owns is based on the value defined in the policy configuration, but the actual value is always adjusted based on the current size of the resource group.

For example, if you configure Consumer X to own 10 slots in a resource group with 100 slots (so X owns 10/100 configured slots), and then you add hosts to the resource group until it actually has 120 slots (without changing the configuration), the quota is calculated assuming that X owns 12 slots, not 10. If some hosts become unavailable, and the resource group has only 90 slots available, the quota at that time is calculated assuming that X owns only 9 slots.

How many slots can the consumer borrow?

The number of slots a consumer is entitled to borrow is proportional to the number of slots it owns. For example, if sibling consumers own 10 and 20 slots respectively, and still need more slots, any additional slots they share will be distributed in a 1:2 ratio.

If sibling consumers own 0 slots, any slots they borrow are shared equally among them.

Calculating the quota with reserve and limit

In more complex cases, reserve and limit affect the quota calculation:
  • The maximum quota is the limit, even if the consumer has demand greater than its limit. The value of the limit does not depend on the size of the resource group.
  • The minimum quota is the reserve (if a consumer reserves slots, the quota can exceed the demand, and slots can be idle). The reserve is not normally adjusted based on the size of the resource group. For example, in a resource group with 100 slots configured, suppose you configure Consumer X to own 10 slots and reserve 6, so X owns 10/100 configured slots and reserves 6:
    • If the resource group doubles in size, or reduces in size to 80 slots, the number of slots owned is adjusted proportionally, but the quota is still calculated assuming X reserves 6 slots.
    • However, if the number of slots in the resource group is reduced to 50, the number of slots owned by X becomes 5. The consumer cannot reserve more slots than it owns, so the quota is now calculated assuming X reserves 5 slots.

Calculating the quota with rank

Rank changes the quota when the mathematical calculation of the quota results in a fraction (the system must allocate a whole slot, or none at all).

For example, if Consumer A owns 1 slot and Consumer B owns 2, any slots they can borrow are shared in a 1:2 ratio. If there are 10 slots, the system cannot really allocate 3.333 slots to A and 6.666 to B, so the final quota depends on rank:
  • If A has the higher rank, its quota of 3.333 is increased to 4, and B's quota becomes 6.
  • If B has the higher rank, its quota of 6.666 is increased to 7, and A's quota becomes 3.

Calculating the quota in a hierarchy

If the consumer tree includes sub-consumers, the system does not compare all the leaf consumers at once. Instead, the quota is calculated among the root-level consumers first, then the slots assigned to each parent are shared among its child consumers.
  • For a parent consumer, the demand is calculated by adding up the demand of each child consumer.
  • Among child consumers, the quota is calculated by dividing up the quota assigned to their parent consumer.

Slot distribution

The quota is the number of slots a consumer is entitled to have. If the number of slots actually allocated to the consumer is different, the consumer is either over-allocated or under-allocated, and the system needs to reallocate slots.

After calculating the new quotas, the system reallocates slots, so each consumer gets the correct amount.
  1. The first step in this process is the distribution of available slots; the system distributes any idle slots to consumers that are under-allocated (compared to their quota). When distributing slots, the system allocates all the available slots to the root-level consumers. If any consumers have sub-consumers, the slots assigned to each parent are then shared among its child consumers. The order of distribution matters because the number of idle slots is not usually enough to meet all the quotas. The first consumers to get slots will get the slots that are available immediately, and the last consumers to get slots will get the slots that have to be reclaimed.
    1. At the root level, distribute all the idle slots to under-allocated consumers until they reach their quota, in order of rank. (Among consumers of equal rank, the system follows the order that they appear in the internal configuration file ConsumerTrees.xml; this corresponds to the order that they appear in the complete consumer tree under the Consumers tab.)
    2. If the consumer is a parent, the slots are passed down to its child consumers, also in order of rank.
  2. The second step in the process is reclaim. If all the idle slots are distributed and some consumers are still under-allocated, the system will reclaim slots.

Reclaim

After calculating new quotas, the system reallocates slots, so each consumer gets the appropriate amount. If the system does not have any idle slots to distribute, but some consumers are still under-allocated (compared to their quotas), it means at least one other consumer is over-allocated.

To reclaim slots from consumers that are over-allocated, the system terminates and requeues any task that is running on the slot. The idle slot is then allocated to another consumer.

Reclaim is initiated at the end of a scheduling cycle, and if the task releases the slot quickly, the slot becomes available for the system to distribute in the next scheduling cycle. The under-allocated consumer that is waiting for the slot waits for at least one scheduling cycle. The actual time to release the slot depends on system overhead, time to clean up after terminating a task, and the reclaim grace period.

Reclaim grace period

By default, there is no reclaim grace period, and a task is killed immediately to reclaim the slot.

The reclaim grace period is intended to improve overall efficiency of resource usage. When a slot must be reclaimed, and a grace period is defined, the task will keep running until it completes, or until the grace period expires, whichever comes first. If the task fails to complete before the grace period expires, it is terminated and requeued.
  • If the task finishes within the time allowed, it releases the slot immediately (instead of waiting for the full grace period).
  • If the task is still running when the grace period expires, the task is terminated and requeued.

You configure the grace period separately for each consumer. The over-allocated consumer (the one running the task) defines the grace period that is used during reclaim. The consumer that needs the slot has no control over how long it takes to reclaim it.

Tip: One way to mitigate this situation is to design a consumer hierarchy that places consumers with the same grace period under the same parent consumer. Since both share the quota of the parent consumer, it reduces the possibility of borrowing and lending among consumers with different grace periods.

Disable reclaim when owned slots are proportionally reduced

When the number of slots the consumer owns has been proportionally reduced, you can stop the system from reclaiming some slots from this consumer.

For example, if you configure Consumer X to own 10 slots in a resource group with 100 slots (so X owns 10/100 slots), and the resource group has only 90 slots available at this time, the quota is calculated assuming that X owns only 9 slots.

Then if X is currently using 12 slots and has a new quota of 9, the system should reclaim 3 slots (based on a quota of 9 slots). However, you can configure the system to reclaim only 2 slots (based on the original configuration of 10 owned slots). In this case, the task running on the third slot will be allowed to finish, no matter how long that takes. Then the slot will become available for distribution to another consumer (based on a quota of 9 slots).

To configure this feature, set EGO_DISABLE_RECLAIM_HYBRID_OWN=Y in EGO_CONFDIR/ego.conf.

Free slots

When at least one consumer has high demand and the consumer is configured to use the hybrid policy, the free slots under the hybrid policy is equal to MAX, which is the (sum of children's FREE slots, hybrid reserved slots minus hybrid allocated slots). For a leaf consumer without sub-consumers, the number of free slots is equal to (hybrid reserved slots minus hybrid allocated slots). If the number of hybrid allocated slots is higher than the number of hybrid reserved slots, the number of free slots is zero.

If none of the consumers has high demand or if there is no demand for any consumer, the number of free slots is calculated proportional to the configured number of owned slots.

Configure the even distribution of free slots to all consumers

If a consumer owns 0 slots, it will only get free slots when there is no outstanding demand by all of its siblings. It is possible to change this default behavior so that slots that are not consumed by its owner can be distributed evenly across all sibling consumers with outstanding demand, regardless of how many slots they own.

When the hybrid scheduling policy is configured to distribute free slots evenly, the following algorithm is used:

Note: The algorithm does not consider hybrid limit and hybrid reserved settings; the behavior of these settings is not affected by this feature.

If free_slots >= total_owned_slots of all consumers, free slots are distributed on each consumer level, as follows:

  1. To each consumer according to MIN(demanded_slots, owned_slots).
  2. To each consumer that still has demand until either the demand is satisfied or the slots are depleted.

If free_slots < total_owned_slots of all consumers, free slots are distributed on each consumer level, as follows:

  1. A portion of owned slots (adjusted_owned_slots) = (available_slots * owned_slots/total_owned_slots) to each consumer, where total_owned_slots is the sum of owned slots of all consumers, including consumers that have no demand.
  2. To each consumer according to MIN(demanded_slots, adjusted_owned_slots).
  3. The preceding steps are repeated until the ownerships of all consumers have been satisfied.
  4. If slots are leftover, distribute them evenly to all consumers that still have demand until either the demand is satisfied or the slots are depleted.
For example, assume we have the following sibling consumers with 30 free slots available:
  • Consumer A owns 15 slots
  • Consumer B owns 15 slots
  • Consumer C owns 0 slots
Then, by default:
  1. Consumer C submits 100 tasks; it is allocated 30 slots.
  2. Consumer A submits 100 tasks; it reclaims all the slots from consumer C.
  3. Consumer A gets all 30 slots, 15 owned plus 15 from consumer B.
  4. Consumer C gets 0 slots.
When configured to distribute free slots evenly to consumers:
  1. Consumer C submits 100 tasks; it is allocated 30 slots.
  2. Consumer A submits 100 tasks; it reclaims its 15 owned slots.
  3. The remaining 15 slots that are not used by its owner, consumer B, are redistributed evenly between consumers A and C:
    • Consumer A gets an additional 8 slots (assuming it gets the extra slot from rounding 7.5 slots) for a total of 23, 15 owned plus 8 from consumer B.
    • Consumer C gets 7 slots in total.

If we update the previous example with 10 free share slots in total instead of 30, consumer A gets all 10 slots and consumer C gets 0 slots.

To configure the hybrid policy to distribute free slots evenly to all consumers, set EGO_HYBRID_EVENLY_DISTRIBUTE_SLOTS=Y in EGO_CONFDIR/ego.conf.

Feature interactions

Exclusive slot allocation

Exclusive slot allocation can coexist with the hybrid policy, but the policy will first apply the rules of the exclusive slot policy before the rules of the hybrid policy.

Mixing ownership and hybrid policies in one resource group

It is possible to configure both ownership and hybrid policies in one resource group, but only one policy can be configured for a consumer and all its children starting from the first level under the root consumer.

In this type of configuration, the number of slots available for the hybrid policy is the balance from the ownership policy:
  • Configure the owned slots in the hybrid policy so the balance at the root level is equal to 0.
  • If the total number of slots in the resource group is less than the sum of all owned slots from both policies:
    • the ownership policy consumers own the exact number configured in the ownership policy.
    • the hybrid policy consumers all have their owned slots reduced proportionally (the configuration of the hybrid policy does not guarantee that the slots can actually be allocated).
  • If the total number of slots in the resource group is more than the sum of all owned slots from both policies
    • the ownership policy consumers own the exact number configured in the ownership policy.
    • the hybrid policy consumers all have their owned slots increased proportionally (but they are constrained by their limit, which does not change).

Switching policies using time windows

Because resource plan changes can take effect when one time window switches to another, a large number of resources could be reclaimed at once. When the policy switches between ownership and hybrid models, resources in use may be reclaimed, even if they are then redistributed to the same consumers.

For the following example, assume a consumer at time window T1 is configured to use the ownership policy and at time window T2, it is configured to use the hybrid policy.
  • If it has lent out slots at T1, the policy will immediately reclaim back all the lent out slots from its borrowers.
  • If it has borrowed slots at T1, the borrowed slots will be reclaimed.
  • If the number of used slots in T1 is more than owned slots in T2, the delta will be reclaimed.

The same thing happens when a consumer tree at time window T1 is configured to use the hybrid policy and time window T2 is configured to use the ownership policy. If the number of used slots in T1 is more than owned slots in T2, the delta will be reclaimed. The borrower's grace period is observed in both cases.

Note: In both cases, slots borrowed with the ownership policy are independent of slots borrowed with the hybrid policy. The borrower does not need to wait for its borrowed slots from one policy to be released first before it can borrow from another policy. If the borrower has a long grace period, it might temporarily use more slots than you configured in the current active policy, and cause other consumers to be under-allocated. The total number of slots used can exceed the limit during the grace period.