This is the second in a series of articles on best practices for writing rules for ODM Decision Server Insights. Please start with the article on the Basics of Rule Writing.
The first article covered when/occurs reactive rules, fired when an event is received, with optional event and entity conditions. This article expands upon that foundation to cover agents and rules that fire some time after an event has been received, and how to use explicit or implicit scheduled events, detect out of order events and how to avoid some pitfalls.

Using the Scheduler

The DSI runtime invokes an Agent when an event is received. However it can also invoke an Agent using a powerful built-in scheduler. This allows an Agent to ask the runtime for a callback after a time period has elapsed.
Before we look at rules, let’s take a look at the basic scheduler capability and Java Agent APIs. The InsightsStarter example includes a simple Java Agent that creates a scheduled callback using procedural code:

public class DemoJavaAgent extends EntityAgent {
    
    private static final String COOKIE = "ordered more than 3 items";
   
    @Override
    public void process(Event event) throws AgentException {
        Purchase purchase = (Purchase) event;
        
        // if the customer has purchases more than 3 items
        if(purchase.getOrderItems().size() > 3 ) {
            // then we schedule a callback in 1 days time
            schedule( 1, TimeUnit.DAYS, COOKIE );
        }
    }
    @Override
    public void process(String key, String cookie) throws AgentException {
        ConceptFactory factory = getConceptFactory(ConceptFactory.class);
        Relationship customerRel = createRelationship((Customer)getBoundEntity());
        CustomerNotification cn = factory.createCustomerNotification( customerRel, "Thank you for your purchase.", ZonedDateTime.now() );
        emit(cn);
    }
}

This code calls the schedule API to request a callback 1 day after a purchase is received for an event with more than 3 order items. When the schedule is triggered the runtime will call the process(key, cookie) method to allow the Agent to perform whatever processing needs to be performed. In this example the Agent emits an event, however it could just as easily update the state of the bound entity.
The equivalent rule looks like this:

when a purchase has occurred 1 days ago
if
    the number of elements in the order items of this purchase
        is more than 3
then
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Thank you for your purchase." ;

Much cleaner and less boilerplate! The use of when/has occurred creates an implicit scheduled callback that will invoke the Rule Agent 1 day after the purchase. When the Rule Agent is invoked 1 day after a purchase all the rules in the Rule Agent will be evaluated, and the rule shown above will fire.
It is important to note that if an entity condition is added to the rule, then this condition is evaluated when the callback occurs, and is based on the state of the entity when the callback occurs, not on the state of the entity with the event is received. For example:

when a purchase has occurred 1 days ago
if
    the number of elements in the order items of this purchase
        is more than 3
    and
        the customer status of 'the customer' is BRONZE
then
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Thank you for your purchase." ;

 
This rule will fire 1 day after a purchase with more than 3 order items in it, if the customer has status of BRONZE, evaluated 1 day after the purchase.

Detecting Missing Events

The scheduler really comes into its own when you need to write patterns of the form:

• “if the customer’s new credit card was posted 7 days ago, and has not been activated, then send the customer an activation reminder.”

This rule is reformulated as:

1. when a credit card dispatch has occurred 7 days ago

2. if there is no credit card activation after this credit card dispatch

3. then emit a new customer reminder;

Line 1 specifies that the arrival of the credit card dispatch event should create a scheduled event 7 days later. Line 2 looks in the event history to see whether a credit card activation event has been received after the triggering credit card dispatch event. If there is no credit card activation event in the event history then the action of the rule is fired, emitting a customer reminder event.
Note that for this rule to function correctly the Rule Agent needs to keep credit card activation events in event history for a minimum of 7 days.
Here is a similar example from InsightsStarter:

when a purchase has occurred 30 days ago , called 'LAST'
if
    the customer status of 'the customer' is GOLD
    and there is no purchase after LAST
then
    set the customer status of 'the customer' to SILVER ;
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "You have been downgraded from gold to silver due to lack of activity.";

You can execute this example and use the Inspector to visualize the events received as well as the scheduled events (callbacks).

What Time is It?

If you’ve been playing with the rule editor you have probably seen there are temporal constructs you can use in rules such as:

• now
• current second / minute / hour / day / week / month / year
• last period of
• next period of

Each Agent in DSI has its own value of now, which is defined as the logical timestamp of the temporally most recent event that has been received by this agent. That is to say, if an Agent receives a sequence of events timestamped April 1st 2015, May 3rd 2015 and June 4th 2015 then the value of ‘now’ for the Agent advances from April 1st to May 3rd to June 4th.  The value of now for an Agent is based on logical timestamp of the events received, not on wall clock time or machine time. This makes it easy to test and simulate how an agent would react to different event sequences, as the agent’s notion of now is determined by the events in the sequence, and not by some external clock. Test scenarios can run at full speed, based on the logical timestamps in the events.

Out of Order Events

Astute readers should be asking themselves what happens if events are received out of order. In the case of out of order receipt of events the value of now is no longer the logical timestamp of the event last received, but is the logical timestamp of the most recent event. So if the agent receives a sequence of events timestamped April 1st 2015, June 4th 2015 and  May 3rd 2015 then the value of ‘now’ for the Agent advances from April 1st to June 4th and remains as June 4th when the last event is received. The value of now can only advance!
The value of now is used to compute the current second, minute, month etc. as you would expect. This allows you to write powerful rules such as the rule below:

when a purchase occurs
if
    the day of week of now is Monday
    and the duration of the current month in days is 30
then
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Thank you for shopping on a Monday in Sept, April, June, Nov.";

The rule has two references to now, one explicit, and one implicit through the use of the current month. Beware of the behavior of now with respect to out of order events: this rule could fire repeatedly for events that did not occur on a Monday, if they are received out of order and the latest event received occurred on a Monday in a month with 30 days.
Contrast the rule above with this one:

when a purchase occurs
if
    the day of week of this purchase is Monday
    and the duration of the calendar month of this purchase in days is 30
then
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Thank you for shopping on a Monday in Sept, April, June, Nov.";

The temporal logic in the rule above is all relative to the logical timestamp of the event just received. The behavior of the rule is therefore easier to understand and test with respect to out of order events, as its behavior does not depend on the logical timestamps of previously processed events. In general avoid using references to now and the current xxx in your rules if possible.
A rule can detect whether an event has been received out of order, because the timestamp of the event will be before now. E.g.

when a purchase occurs
if
    this purchase is before now
then
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Thank you for your out of order purchase.";

This allows your rule logic to potentially include special handling, or compensating logic, for events that are received out of order.
Note that references to now and the current xxx also create implicit time triggers, so should be used very carefully and cautiously as they can have a performance impact. If you do create rules that refer to now or the current xxx you should test them extensively (paying particular attention to out of order events) and use the Inspector to understand the scheduled events that are created, and the expected event load on the system.

Now and Scheduled Events

As we’ve learned Agents use the logical timestamps of received events to determine the value of now. There is one exception however. What if no event is received?
If we write a rule of the form:

when a purchase has occurred 30 days ago , called 'LAST'
if
    the customer status of 'the customer' is GOLD
    and there is no purchase after LAST
then
    set the customer status of 'the customer' to SILVER ;
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "You have been downgraded from gold to silver due to lack of activity.";

When a purchase event P1 is received it will typically advance the value of now, unless the purchase event is received out of order. However the purchase event also creates a scheduled event (callback) for 30 days after the purchase (P1 + 30 days). If no other events are received we clearly want the rule to be re-evaluated 30 days after P1. In the absence of any events this time period is based on the system time of the machine (aka wall clock time).
If however the agent receives another purchase event P2 before the P1 + 30 days callback is fired, and P2 has a logical timestamp that is after purchase + 30 days, then the agent runs and immediately evaluates the rule that created the P1+30 days callback. This ensures that when testing at full speed scheduled events (callbacks) work as expected.
If your tests need to manually control the passage of time you can use TestDriver.processPendingSchedules or continue processing until in your Test Scenarios. These methods will force pending schedules to be fired, allowing you to manually advance time and check the state of the system. You only require these methods if you cannot send events that also advance time and cause time triggers to be re-evaluated.

Last Period of Gotcha

One common gotcha that has befallen many DSI rule writers is the use of last period of xxx to filter events. For example:

when a purchase occurs, called LAST
if
    the customer status of 'the customer' is BRONZE
        and there are at least 5 purchases during the last period of 30 days
then
    set the customer status of 'the customer' to SILVER;
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Congratulations you are a silver customer.";

This rule is trying to count the number of purchases in the past 30 days. There are two potential issues here:

1. last period of is implicitly relative to now, so you need to be carefully test the behavior with respect to out of order events
2. last period of does NOT include now — i.e. it will count all events within the last 30 days, but not including the current event, unless the current event has been received out of order (logical timetamp of the current event is less than now).

To fix both these issues you can filter events using the timestamp of the current received event as an anchor:

when a purchase occurs, called LAST
if
    the customer status of 'the customer' is BRONZE
        and there are at least 5 purchases after 30 days before LAST
then
    set the customer status of 'the customer' to SILVER;
    emit a new customer notification where
        the customer is 'the customer' ,
        the message is "Congratulations you are a silver customer.";

And count all events that occur after 30 days before the event just received (LAST). This will include the event just received.