Decision Server Insights – Technical Overview
In this article we would like to provide a technical overview of Decision Server Insights, including lots of links to examples and reference material including product documentation to help you quickly get up to speed with this exciting new member of the ODM family.
This article is based on Decision Server Insights v8.8.
Key concepts – Events, Entities, Agents and Context
Events
An event for Decision Server Insights is a structured (machine readable) description of something that has happened in the real-world. It has a time stamp within it which encodes the time of occurrence of the event. Events have an arbitrarily complex structure and may be represented as instances of an XML Schema.
Business Entities
Events that have occurred are typically correlated with a business entity. For example, “a change of address event is related to a customer“. The incoming change of address event contains the identifier of the customer who has changed their address. Entities have an arbitrarily complex structure and may be represented as instances of an XML Schema. Entities have a single identifier (a string) that is used to identify the entity.
Agents
Agents are units of business logic that subscribe to events and may detect situations. In many cases they are also responsible for updating the state of a business entity. For example, the customer agent may subscribe to change of address events and will set the address of the customer based on an incoming change of address event. In addition however it can examine the history of change of address events for a customer and detect suspicious patterns that might indicate Account Takeover Fraud has taken place.
Agents may be implemented using business rules or by Java code.
Context
DSI can collect context information from a wide range of sources and events over a period of time.
An agent runs within a context which it uses to detect situations. The context includes:
- the state of its bound entity, if applicable. For example, the Customer Agent is bound to a Customer entity and will have access to all customer data.
- for rule-based agents, the event history, i.e. all the events correlated with a given business entity.
- spatial information related to the incoming event, the bound entity and the event history
- scores calculated by predictive scoring services based on the state of the entity and incoming events
Read more from the DSI product documentation introduction.
Modeling
When creating a DSI solution a developer typically creates a Business Object Model (BOM). The BOM defines the structure of the events and entities for a given business domain. The BOM may be created from existing XSDs, event definitions in IBM Integration Bus. DSI has wizards to make this easy
It can also be defined in Insight Designer using the Business Model Definition language: a textual domain specific language for defining DSI BOMS.
Events and Entities modeled using DSI can be serialized to XML, using either a generic XML Schema or a strongly-typed XML Schema that is generated from the BOM.
Business Vocabulary
A business vocabulary for a target locale (currently only English is supported) is automatically generated from business model definition files, or may be explicitly specified by editing the BOM. The business vocabulary defines the terms and phrases which will be used when writing rules and aggregates.
Developer Tooling
DSI Solutions are created and edited inside Insight Designer, an Eclipse-based development environment. The development environment can be extended with plugins to manage the DSI runtime, to start and stop the server for example.
Governance
The artifacts within a DSI solution are stored as files within projects within an Eclipse workspace. The solution developer can use Eclipse Team Provider plugins to manage the artifacts as they would manage other source code.
Business User Tooling
DSI does not currently integrate with Decision Center.
End User Tooling
End User tools, e.g. tools that would allow a non-technical user to parametrize pre-existing rule templates, can be built using the DSI REST APIs.
Testing Solutions
DSI solutions are tested can be tested using Test Scenarios, Event Sequences and Entity Loaders. These artifacts allow you to use high-level controlled natural language to define sequences of events, entities and checks (assertions).
JUnit test cases, that use the TestDriver APIs to submit events. Assertions may be created over the state of entities after events (or a sequence of events) have been submitted, as well as any events emitted by individual agents. The Inspector, a web-based timeline view of agent execution, may be used to view recordings of event processing.
Server Management and Monitoring
Shell scripts are provided for all server management tasks, including: creating servers from a template, deploying solutions, managing connectivity, and disaster recovery. JMX MBeans are used to expose key performance statistics and operations.
A web-based console is provided to monitor a group of DSI servers.
Submitting Events for Processing
Inbound events are submitted to the DSI runtime as XML documents, or calls to the SolutionGateway Java API. The DSI runtime can listen on JMS queues or topics, or events may be submitted by performing an HTTP/S POST to a URL specified for a solution.
Triggering Actions
When an Agent detects a situation of interest it emits an event. The agent’s solution can be configured to place the emitted event on a JMS queue or topic, or POST it to a URL. The event is serialized to a XML document that conforms to the XML Schema for the event type. External actions are triggered by either routing the event to an ESB or Broker, or by writing a Message Driven Bean that consumes the event and performs the necessary actions.
Correlating Events
DSI correlates incoming events with an entity of interest (the “bound entity” for an agent). This allows the platform to route all incoming credit card events for the account “123” to an agent instance for the given account.
In addition rules within a Rule Agent may perform more fine-grained correlation between the events received by the agent. For example, a bonus may be granted to a customer that sends a purchase event and has previously opted-in for a marketing offer (an event).
An agent subscribes to a purchase event:
‘demo_rule_agent’ is an agent related to a customer,
processing events :
– purchase, where this customer comes from the customer of this purchase
The agent processes the purchase event in the context of a given customer entity ‘the customer’:
when a purchase occurs, called LAST
definitions
set ‘RECENT PURCHASE’ to a purchase
where this purchase is during the last period of 28 days ;
set ‘RECENT TOTAL’ to the total amount of all order items
in the order items of ‘RECENT PURCHASE’ ;
if
the customer status of ‘the customer’ is SILVER
and ‘RECENT TOTAL’ is more than 300
and there are at least 3 purchases after 14 days before LAST
then
set the customer status of ‘the customer’ to GOLD;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Congratulations you are a gold customer.”;
Aggregating Events
The rules within a Rule Agent may easily count, filer and aggregate the events received.
when a purchase occurs, called LAST
if
the customer status of ‘the customer’ is BRONZE
and there are at least 5 purchases after 30 days before LAST
then
set the customer status of ‘the customer’ to SILVER;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Congratulations you are a silver customer.”;
Using Entity State
Java agents and rule agent both have access to an optional bound entity — allowing them to check the current account balance for example, when processing credit card transaction events.
when a purchase occurs, called LAST
if
the total amount of all order items
in the order items of LAST is more than 100
and the customer status of ‘the customer’ is NONE
then
set the customer status of ‘the customer’ to BRONZE;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Congratulations you are a bronze customer.”;
Using Shared Aggregates
Shared Aggregates allow you to aggregate events related to an entity. They can provide important performance advantages as the full events are not retained, just the summary aggregated statistics.
the total transaction amount of a customer is aggregated from transactions ,
where this customer comes from the customer of each transaction
as the total amount of all transactions
available for 365 days with a resolution of 6 hours.when a transaction occurs
definitions
set AMOUNT to the total transaction amount of ‘the customer’
over the period between the start of the calendar day of this transaction
and the end of the calendar day of this transaction;
if
AMOUNT is more than 100
then
define ALERT as a new alert where
the customer is ‘the customer’ ,
the message is “The customer has spent ” + AMOUNT + ” on the day of ”
+ the timestamp of this transaction;
add ALERT to the alerts of ‘the customer’;
emit ALERT;
Global Aggregates
Aggregates over the entire set of events received by a DSI runtime, or over the entire set of business entities, may be defined. These global aggregates may then be used within rule and Jave agents to relate the behavior of an individual entity with the typical behavior of an entire population of entities. Aggregates may use filtering conditions (including time):
define ‘average order amount’ as the average amount of the order items of all orders ,
where the timestamp of each order is during the current week,
evaluated every day at 9:28:58 AM
when a purchase occurs, called LAST
if
the total amount of all order items
in the order items of LAST is more than ‘average order amount’
and the customer status of ‘the customer’ is NONE
then
set the customer status of ‘the customer’ to BRONZE;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Congratulations you are a bronze customer.”;
Temporal Rules
Rules often make use of sliding or tumbling time windows, collecting events over the recent (or not so recent) past and aggregating and filtering.
when a purchase occurs, called LAST
definitions
set ‘RECENT PURCHASE’ to a purchase
where this purchase is during the last period of 28 days ;
set ‘RECENT TOTAL’ to the total amount of all order items
in the order items of ‘RECENT PURCHASE’ ;
if
the customer status of ‘the customer’ is SILVER
and ‘RECENT TOTAL’ is more than 300
and there are at least 3 purchases after 14 days before LAST
then
set the customer status of ‘the customer’ to GOLD;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Congratulations you are a gold customer.”;
Running Agents on a Schedule
Java agent and rule agents may request a callback at a future time, allowing them to schedule work that should be performed in the future. Using rule agents these schedules are created implicitly based on the semantics of the rules written and their references to the ‘now’ parameter.
Detecting Missing Events
Schedules may be used to detect missing events, for example, blocking a credit card if it has been sent to the customer and has not been activated for the past 7 days. The rule below is triggered 30 days after the customer has completed a purchase and can look back in time to see if any purchases have been made since. If no purchase events have been received since the last purchase, and the customer has GOLD status, then the customer is downgraded to SILVER status.
when a purchase has occurred 30 days ago, called ‘LAST’
if
the customer status of ‘the customer’ is GOLD
and there is no purchase after LAST
then
set the customer status of ‘the customer’ to SILVER ;
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “You have been downgraded from gold to silver due to lack of activity.”;
Date and Time Handling
DSI has a rich and powerful library of date and time data types, based on ISO 8601 and JSR-310. These data types are exposed through the rule language allowing you to create rules that use time instants, durations and time periods. The rule language and underlying algorithms are time zone aware and take into account Daylight Saving Time when necessary.
Spatial Data Types
DSI has a built in business object model and vocabularly for spatial data types and algorithms. The can be used to create rules of the form:
when a purchase occurs
definitions
set ‘PURCHASE LOCATION’ to the location of this purchase;
set ‘HOME LOCATION’ to the home location of ‘the customer’;
if
the distance between ‘HOME LOCATION’
and ‘PURCHASE LOCATION’ in kilometers is less than 5
then
emit a new customer notification where
the customer is ‘the customer’ ,
the message is “Enjoying local shopping?” ;
Building Dashboards
Entities and global aggregates are exposed transparently over a REST API making it easy to build custom dashboards that display DSI data. In addition other monitoring tools, such as Business Monitor may be used to create graphical dashboards based on the events emitted by DSI.
Visualising Geospatial Entities
You can use the built-in Map Viewer to automatically visualize entities with geospatial attributes on a map.
Referencing Systems of Record
Entities may be initialized from data held in external systems of record by using Java Initializers.
Modeled entities may contain fields that are enriched by external data providers. Data Providers are Java classes that call external systems of record. The DSI platform can optionally cache the results from Data Providers to reduce the performance impact on external systems.
Declaration of a Data Provider:
a geocoder is a data provider,
accepts a street, a city, a country, a zip,
returns a location (a point).
Using an initializer to create an entity from an event
a customer is initialized from a customer registration , where this customer comes from the customer of this customer registration :
– set the first name of this customer to the first name of this customer registration
– set the last name of this customer to the last name of this customer registration
– set the dob of this customer to the dob of this customer registration
– set the customer status of this customer to NONE
– set the home address of this customer to the home address of this customer registration .
Using a Data Provider to enrich an entity:
a customer is enriched by the geocoder,
given
the street from the street of the home address of this customer,
the city from the city of the home address of this customer,
the country from the country of the home address of this customer,
the zip from the zip of the home address of this customer,
setting
the home location of this customer to the location of this geocoder.
Operationalizing Predictive Models
DSI includes wizards to integrate calls to SPSS Scoring Server during event processing. This allows the results from a scoring model (e.g. customer churn) to be used as part of the context available for situation detection. You can either create a Predictive Scoring Agent or create a Predictive Scoring Data Provider.
Processing Events using Java Code
Developers may write Java classes (Java Agents) that subscribe to events processed by DSI. Java Agents are useful when integrating with existing libraries, external systems or when the flexibility of a general purpose programming language is required, e.g. for complex calculations that cannot easily be performed using rules. Java Agents have many of the same capabilities of Rule Agents, including the ability to read global aggregates and schedule callbacks.
package demo_solution.demo_java_agent;
import java.util.concurrent.TimeUnit;
import org.acme.*;
import org.threeten.bp.ZonedDateTime;import com.ibm.ia.agent.EntityAgent;
import com.ibm.ia.common.AgentException;
import com.ibm.ia.model.Entity;
import com.ibm.ia.model.Event;
import com.ibm.ia.model.Relationship;public class DemoJavaAgent extends EntityAgent<Entity> {
private static final String COOKIE = “ordered more than 3 items”;
@Override
public void process(Event event) throws AgentException {
Purchase purchase = (Purchase) event;
// if the customer has purchased more than 3 items
if(purchase.getOrderItems().size() > 3 ) {
// then we schedule a callback for 1 day in the future
schedule( 1, TimeUnit.DAYS, COOKIE );
}
}@Override
public void process(String key, String cookie) throws AgentException {
ConceptFactory factory = getConceptFactory(ConceptFactory.class);
Relationship<Customer> customerRel = createRelationship((Customer)getBoundEntity());
CustomerNotification cn = factory.createCustomerNotification( customerRel, “Thank you for your purchase.”, ZonedDateTime.now() );
emit(cn);
}
}
Non-Functional Requirements
Transactional Guarantees
DSI guarantees that if an event has been accepted for processing then it will be processed at least once. In addition an event is either wholly processed (by all subscribed agents) or not at all. On an agent failure the entire event processing transaction is rolled-back and no side-effects are preserved within the system.
Scalability
A well designed DSI solution scales linearly, allowing the data and compute capacity of the system to be expanded by adding more machines to the grid.
Elasticity
The DSI data/compute grid is designed to expand elastically without shutdown and will dynamically repartition data and distribute computation to make use of increased or reduced resources.
High-Availability
DSI is designed to be highly available under a wide range of software, hardware and network failures. DSI can replicate all state synchronously and asynchronously to a user defined number of replicas within the grid. In addition the grid can be synchronously or asynchronously written to a DB2 instance.
Continuous Availability
A DSI grid does not need to be shutdown for solution upgrades, software maintenance or hardware maintenance. DSI supported rolling upgrades of its underlying software and hardware and new solution versions can be deployed easily and with no downtime.
Disaster Recovery
DSI may be configured to write grid state to a DB2 database (which may in turn be replicated across data centers). A DSI grid can be rapidly restored from a DB2 disaster recovery database.
Performance Monitoring
DSI includes JMX MBeans for performance monitoring — allowing the performance cost of individual agents, or event types, to be retrieved.
Performance and Sizing
DSI solutions can be challenging to size, based on the wide variety of situation detection patterns that can be operationalized and the number of non-functional configuration parameters supported. Your account team can help and advise with sizing.
Topology
DSI is packaged with its own runtime, which is based on WebSphere Liberty Profile. The data/compute grid fabric for DSI is based on IBM WebSphere eXtreme Scale (WXS).
DSI supports everything from a single JVM running on a laptop (useful for developers building solutions locally) all the way up to data/compute grids composed of hundreds of machines. In general a small number of machines are dedicated to inbound JMS/HTTP, a variable number of machines are running agents (business logic), a small number of machines are catalog servers (maintaining the structure of the grid), while a small number of servers or handling outbound JMS/HTTP messaging.
Hardware and OS Support
DSI runs on 64-bit RHEL Linux, SLES, AIX, System Z and Windows. Virtualization may be used if necessary.
Please refer to the official system requirements page for details.
[…] View the full article […]
[…] Infographics Courtesy: Kolban’s Book on IBM Decision Server Insights & IBM […]