IBM Operational Decision Management (IBM ODM), the next generation of business rules, is a combined rules and events management platform designed to automate and govern frequently occurring, repeatable business decisions across critical business systems. It provides:
dW Live! IBM Operational Decision Management
JamesLove 270005E1QB 610 Views
For more information and your free download please visit: http://www.ibm.com/developerworks/downloads/soasandbox/odm.html
JustinPhillips(BP3) 2700065DAF Tags:  testing odm realtime champion-challenger simulations 714 Views
This tutorial outlines how to implement a simple mechanism within a decision service to allow real-time champion challenger or multi-variate testing within a live, production environment.
Why not use DVS to test changes?
ODM comes packaged with a testing framework called Decision Validation Framework (DVS). The purpose of this framework is to allow users of Decision Centre to test rule changes against predefined scenarios.
For testing Key Performance Indicators (KPI’s) against an historical data set, the simulation capabilities in DVS also offers a big advantage at design time. You can see how a change in the rules could have an impact on different aspects of your decision service. For example, how many customers could be accepted or declined before and after the rule change.
However, this only gives the owners of the rules a perspective on how the rules work from a historical data point of view. What if you want to test for example customer purchasing behaviours in real-time, today? Providing a real-time method of testing variations allows the business an instant method of testing different data profiles in a safe and controlled manner.
When is it safe to use this method of testing?
Why should this not be a safe method of testing? The danger with real-time testing is that you will not know the effects of a rule change until the decision service has been executed many times to provide a large sample to gauge whether the change has had a positive or negative affect.
So in order to use real-time testing of decisions within a production environment in a safe and controlled way, a couple of things should be put in place:
Having these components in place will mean that the business users can safely test radical rule changes in real-time, with minimal risk to the business.
How does it work?
This method uses different variations of rules within a decision service, which are packaged and distinguished from each using rule packages. Each variant package will typically contain rules that are a copy of each other (but does not have to be), with slight variations. For example, the logic of the rules maybe the same, but the values being tested against used may differ.
The following sections describe the component parts that make up the multi-variate mechanism. This is based on a sample project that can be downloaded at the end of this article.
The whole process is controlled using a rule flow that directs the request down to the rule variant packages, depending on the randomly selected variant.
The diagram below shows a sample rule flow that has three rule variant packages. The “Control rules” package contains the weighing decision table, and the “Get random variant” action task calls the random distribution XOM method.
Weighting controller decision table
The weighting distribution between the different variants is controlled using a decision table. This table allows the user to set the percentage weighting, based on certain conditions. These conditions allow the weighting to be specifically tailored depending on for example, the income range of the customer, as shown in the example below. However these conditions can be as complex as you require.
The variants are set in the action columns by calling a Java XOM method that passes in the variant type (e.g. A, B or C), and the percentage value. The total percentage across all three variants must add up to 100% (an enhanced version of this mechanism could implement a customisation to check the total values add up to 100% when the table is saved).
The screen shot below shows the BAL used in the action column to set the percentage weighting for a given variant.
In the sample project, there is an alternative control rule that uses an action rule with no conditions. This can be used if you don’t have any need to tune the variants based on the state of the input data, in this case, the customer’s income band. To test this out, just replace the “Variant percentage allocation” decision table rule in the “Control rules” package with the alternative rule in the “Alternative control rules” package.
Random variant generation
The variant is selected randomly based on the configured weighting. This is done within a Java XOM class, invoked from the “Get random variant” action task in the rule flow.
This means that for example if variant A is set to 50%, then there is a 50% percent chance that the randomly generated number will fall within its boundary and so on. The table below shows how this works based on row 2 of the control decision table in the sample project.
So in this example, if a random number value of 55 is generated, then Variant A rules are executed; 71 then Variant B; 100 Variant C.
The code below taken from the VariantCalculator.addVariant() method shows how the variants are added and how the boundaries are calculated. As each variant is added, the total percentage count is calculated to ensure that the variants added do not exceed 100%.
public void addVariant(VariantType variantType, int percentage)
// The sum of all the variants added cannot exceed 100%
Once the variants have been added, then the variant can be randomly selected, using java.security.SecureRandom. This ensures that the selection pattern for variants is non-deterministic, over using the standard java.util.Random algorithms. The performance of SecureRandom will be more than sufficient for high volume calls.
Within the “Generate random variant” action task in the rule flow is a BAL statement that is bound to the VariantCalculator.getRandomVariant() method. When this is invoked, it generates a random number between 1 to 100. The variants and the boundaries are then iterated through, and depending on where the random number falls within the variant boundaries, that variant is returned back. The variant is then used to direct the execution to a rule package in the rule flow. The code below taken from the VariantCalculator.getRandomVariant() method.
VariantType variantType = null;
Iterator<int> percentageIter = variantPercentageMapping.keySet().iterator();
The variant type uses an enumeration so that the correct domain values are automatically available in the control decision table cells, which ensure that the users can only enter correct values for the variants.
The return message from the decision service in this example contains the variant type selected, as well as the selected product, which differs depending on the variant selected. The variant type can be used in the reporting mechanism.
Testing the service
To test the decision service, the next step is to deploy it to a Decision Server as a HTDS service, and using a test harness such as SoapUI, you should see the random variations being returned. Have a play around with the weightings and run more tests, and you will be able to see the distribution change.
How to report the real-time effects of the changes?
Once the decision service deployed and being executed, how can the returned data be reported on to measure the success of the variant?
The reporting mechanism could be provided by BI reports on top of a real-time data warehouse, fed by ODM’s Decision Warehouse. Joined with other customer or operational data, a powerful insight into complex customer behaviours can be gained. These can then be used to tweak the product variations or test out new radical configurations, and provide a feedback loop back into the decision making process.
You can find the sample project used in this article here (champion-challenger.zip).