Grace periods

There are two grace periods that can be configured for the multidimensional scheduling policy: reclaim and Lendout.

Reclaim grace period

The reclaim grace period of the borrowing consumer takes effect when the resource that the consumer is using is being reclaimed. The reclaim grace period is defined for each consumer globally, independent of the resource plan.

If your applications cannot handle failures due to thread interruptions from resource reclaims, disable resource reclaim for the Spark instance group. To disable resource reclaim:

Configure the SPARK_EGO_RECLAIM_GRACE_PERIOD parameter during Spark instance group configuration. The value must be higher than the length of time to complete the workload.
Configure the reclaim grace period in the consumer properties for the consumer that the Spark instance group belongs to. The value must be higher than the length of time to complete the workload.

Lendout grace period

Lendout describes the process of an owner lending out resources to a share pool. If a consumer lends out resources, the Lendout grace period is the maximum duration that the borrower's reclaim grace period can be. Consequently, if a borrower's reclaim grace period is longer than the Lendout grace period, the borrower cannot borrow the resource. The Lendout grace period is defined per consumer per resource plan.

Note: A borrower can only borrow resources when the reclaim grace period is longer than the Lendout grace period if the EGO_MDS_LENDER_RECLAIM_GRACE_PERIOD parameter is enabled. See ego.conf reference.

Resources can be lent out to a consumer's parent and, in turn, the parent can lend the resources to its parent, and so on. For a resource owned by a leaf consumer and lent out to a share pool, all middle-level consumers must enable Lendout. The effective Lendout grace period is the shortest Lendout grace period of all consumers in the path from owner to the node whose highest-level pool borrowed the resource. A consumer cannot borrow the resource if its reclaim grace period is longer than the effective Lendout grace period. For example, in the following diagram, the grace periods of the nodes along the red path are used to determine the effective Lendout grace period. The grace periods of the green path for the borrower's grace period are not considered. The reclaim grace period of the borrower consumer B is compared to the effective Lendout grace period and if the reclaim grace period is less than or equal to the effective Lendout grace period, consumer B can borrow the resource.

Share pool grace period

Grace periods can be defined for both private (local) share pools and public share pools. Any consumer using the share pool must have a reclaim grace period that is equal to or less than the share pool grace period. A private share pool can be used by the descendants of the node and other consumers but other consumers can only use the share pool if it lends resources out, in which case, the Lendout grace period takes effect and the share pool grace period is not considered.

The grace period for private share pools is defined at the consumer level by PrivatePoolGracePeriod. The grace period for public share pools is defined at the resource plan level by DefaultGracePeriod; this grace period also applies to consumers that do not have their own grace period defined and cannot inherit it from their parent (their parent also does not have a grace period defined).

Configure the grace period for private share pools

The grace period for private share pools is defined by PrivatePoolGracePeriod in the OwnershipPolicy element for each consumer in MDPlan.xml. To configure the grace period for private share pools, edit the MDPlan.xml file at $EGO_CONFDIR, as follows:

<OwnershipPolicy PrivatePoolGracePeriod="100">
          <ResourceGroup Name="MDSHosts" PreferenceLevel="5" GetFreeFromNextLevelBeforeReclaim="Y">
            <HostSelection Type="NumHosts">
              <NumHosts Type="absolute">0</NumHosts>
            </HostSelection>
            <Lendout GracePeriod="10"/>
          </ResourceGroup>
</OwnershipPolicy>
...

The value for the grace period is expressed in seconds. If the grace period is not configured, the consumer will inherit the grace period of its parent, or inherit the grace period of the root node and use the DefaultGracePeriod.

Configuring a default Lendout grace period

If the Lendout grace period is not set, the plan level default grace period is used. To configure the default grace period for share pools and any Lendout resources that are not configured separately, edit the MDPlan.xml file at $EGO_CONFDIR, as follows:

<DistributionTree DistributionTreeName="MD1" Type="MDPlan"> 
         <ResourceGroupName> RG1 </ResourceGroupName> 
         <ResourceGroupName> RG2 </ResourceGroupName> 

         <ConsumableResources> 
             <ConsumableResource Optional="true" DefaultConsumption="0">cpu</ConsumableResource> 
             <ConsumableResource>mem</ConsumableResource> 
         </ConsumableResources> 

         <PolicyParameter name="DefaultGracePeriod"> 300 </PolicyParameter>
...

The value for the grace period is expressed in seconds. If the default grace period is not configured, the system default is 0.