IBM technology in the financial markets front office, Part 2: Invoke conditional business rules based on analyzing data streams

Integrating IBM InfoSphere Streams and IBM WebSphere ILOG JRules

This second article in the series focuses on the integration of InfoSphere™ Streams and WebSphere® ILOG® JRules. InfoSphere Streams is a high-performance stream processing engine that is capable of performing calculations and analysis of data streams in real time. ILOG JRules is a business rules management system that enables the creation of rule-based applications. This article presents a simple, algorithmic trading scenario in which InfoSphere Streams operates on a stream of market data and, under certain circumstances, a business rule is invoked using ILOG JRules. In order to perform this task, you will learn how to integrate the two products together in an efficient manner.

Share:

Nick Schofield (nschofie@ca.ibm.com), Software Developer, IBM

Photo of Nick SchofieldNick Schofield is a Software Developer working in the IBM Software Services for WebSphere organization out of the IBM Software Development Lab in Toronto, Ontario. He has worked on a variety of projects including performance testing/tuning, application development, and product integration.



Mary Taylor (marytaylor@us.ibm.com), Consulting IT Specialist, IBM

Author Photo: Mary TaylorMary has been with IBM for over 25 years. She has held a variety of positions, including programmer, systems engineer, project lead, DBA, and data quality specialist. She has been in SWG Technical Strategy for the last four years. Much of that time was focused on Software as a Service. She now administers the SWG Incubation Program. For the last 18 months she has lead a project called Botticelli, which is focused on positioning IBM middleware in the financial markets front office.



05 August 2010

Also available in Chinese Russian Portuguese

Introduction

This article is the second in a series that explores how IBM middleware capabilities can be integrated to address the technical requirements of the financial markets front office. The front office requirements necessitate the need for specialized software that can handle high volumes of data at extremely high speeds. This article series describes an algorithmic trading scenario that was implemented as part of an IBM Software Group incubation project.

InfoSphere Streams

InfoSphere Streams (Streams) is an application development environment and runtime that enables you to create applications that operate on live streams of data in real time. A Streams application is defined in a high-level stream processing language. Each Streams application is a series of operators that perform a function or calculation on the data that flows through it. These operators are then chained together in order to perform complex functions. Many operators that perform functions are included, such as data aggregation, joining, splitting, and complex functions. Streams also enables you to create custom operators to perform specific instructions. These custom operators can be coded in Java or C++, and they are called user defined operators (UDOPs). This article describes how to create a Java UDOP to integrate with ILOG JRules.

WebSphere ILOG JRules

ILOG JRules (JRules) is a business rules management system (BRMS) that enables you to design, develop, and deploy business rules that are accessible from external systems or applications. Business rules are developed using the Rule Studio development environment or the Rule Team Server (RTS) web application. Rules are then deployed to the Rule Execution Server (RES). Applications can connect to the RES and have the rules execute there or acquire the rules and execute them locally. The scenario in this article has the Streams Java UDOP acquire the rules and execute them locally for performance reasons.

Prerequisites

In order to complete this example scenario, you need to have the following software components installed:

  • ILOG Rule Studio Development Environment
  • ILOG Rule Team Server
  • ILOG Rule Execution Server
  • InfoSphere Streams

About the scenario

This article describes an example scenario in which JRules and Streams are required to work together. You can follow the scenario to make it work in your own environment.

The scenario starts with a Streams application that receives stock trades and quotes. You will then perform some simple calculations on this market data to generate a value called a bargain index. If this value is above a certain threshold, you will automatically place an order for this stock. In order to make sure you don't place any large orders without approval, you will modify the Streams application to route any large order requests to JRules first. JRules will then decide if an order should go through automatically or not.


Creating Business Rules using ILOG Rule Studio Development Environment

Complete these steps to create Business Rules using ILOG Rule Studio Development Environment.

Create the XOM

The first thing you do when creating a JRules application is to define the execution object model (XOM). The XOM defines the input and output of the calls you will be making to JRules. For the scenario, you use plain old Java® objects (POJOs) as the interface. It is also possible to use an XML schema to define the XOM.

Complete these steps to create the XOM open Rule Studio using POJOs:

  1. Create a new Java project by entering OrderDecision-XOM for the Project name, as shown in Figure 1, and click Next.
    Figure 1. OrderDecision-XOM project creation
    Screen cap: New Java Project window with OrderDecision-XOM in Project Name field
  2. Create a new Java class in the com.ibm.orderdecision package by entering OrderDecision in the Name field, as shown in Figure 2.
    Figure 2. OrderDecision Java class
    Screen cap: New Java Class window with OrderDecision in Name field
  3. Add the following fields in the Interfaces field for the OrderDecision class, and click Finish.
    • private Boolean canContinue;
    • private String symbol;
    • private Double price;
    • private Double qty;
  4. Create getters and setters for the four fields by following the method shown in Listing 1 for the symbol field.
Listing 1. Create getters and setters
	    public String getSymbol() {
	        return symbol;
	    }
	    public void setSymbol(String symbol) {
	        this.symbol = symbol;
	    }

The canContinue field contains the result of the rule execution and identifies whether the order is an exception or not. The value for canContinue is 0 when the order is an exception. Symbol, price, and quantity are passed into ILOG from the market data that is processed in the Streams program.

Create the rules

Complete these steps to create the actual business rules that will be executed:

  1. Create a new standard rule project by entering OrderDecisionRules for the Project name in the New Rule Project window, as shown in Figure 3.
    Figure 3. OrderDecisionRules project creation
    Screen cap: New Rule Project window with OrderDecisionRules as Project name
  2. Open the Rule Project Map view, and click Import XOM in the Design entity, as shown in Figure 4.
    Figure 4. Import the XOM
    Screen cap: Import XOM highlighted
  3. Select Java Execution Object Model.
  4. Check OrderDecision-XOM, and click OK.
  5. Click Create BOM (Business Object Model) in the Design entity of the Rule Project Map view. The BOM contains the translation of the input Java or XML code to the business term used in the rules engine, as shown in Figure 5.
    Figure 5. Design Entity - step 1
    Design entity with Import XOM (1) and Create BOM highlighted
  6. Accept the defaults on the first page, and ensure the com.ibm.orderdecision package is checked under Select classes on the New BOM Entry window, as shown in Figure 6.
    Figure 6. BOM entry
    Screen cap: New BOM Entry window with checkmark to left of package name
  7. Accept the defaults on the third page, and click OK.
  8. Click Define Parameters in the Design entity in the Rule Project Map view, as shown in Figure 7.
    Figure 7. Design Entity - step 2
    Design entity with Import XOM (1), Create BOM (1), and Define Parameters highlighted
  9. Add the ruleset parameters shown in Table 1.
Table 1. Ruleset parameters
Name Type Direction Default Value Verbalization
symboljava.lang.StringINsymbol
pricedoubleINprice
qtydoubleINquantity
canContinueBooleanOUTfalsecan continue
  1. Click Add Rule Package in the Orchestrate entity.
  2. Click Add rule package to add the ruleset parameters, as shown in Figure 8.
    Figure 8. Orchestrate entity to add rule package
    Orchestrate entity with Add Rule Package highlighted
  3. Name the package com.ibm.orderdecision.
  4. Click Add Ruleflow in the Orchestrate entity, as shown in Figure 9. A ruleflow is a representation of the application business logic. Ruleflows are used to orchestrate rule execution by grouping rules into a series of tasks.
    Figure 9. Orchestrate entity to add ruleflow
    Orchestrate entity with Add Rule Package(3) and Add Ruleflow highlighted
  5. Set the ruleflow options as follows:
    1. Enter /OrderDecisionRules/rules as the Source folder.
    2. Enter com.ibm.orderdecision as the Package.
    3. Enter OrderDecisionRuleFlow as the Name.
    4. Enter RuleFlow as the Type.
    5. Click Finish.
  6. Click Add Business Rule in the Author entity in the Rule Project Map view, as shown in Figure 10.
    Figure 10. Author entity to add business rule
    Screen cap: Author entity with Add business rule highlighted
  7. Set the business rule properties as follows:
    1. Enter /OrderDecisionRules/rules as the Source folder.
    2. Enter com.ibm.orderdecision as the Package.
    3. Enter Order Decision Rule as the Name.
    4. Enter ActionRule as the Type.
    5. Click Finish.
  8. Set the code of the rule as shown in Listing 2.
Listing 2. Order decision rule
if
     price * quantity is less than 500 or symbol is "IBM"
then
     set 'can continue' to true;

This sets the canContinue flag to true for all IBM orders and other orders for which the price times the quantity is less than 500. These orders in Streams are non-exception orders (or straight-through-processing orders).

  1. Edit the OrderDecisionRuleFlow you created earlier.
  2. Enter a starting element, an ending element, and a rule task element on the palette, and join them together as shown in Figure 11.
    Figure 11. Create the rule flow diagram
    Green dot for start, which points to task_1, which points to a red dot for end
  3. Change the ID under Rule Task to Order Decision.
  4. Add the rule com.ibm.orderdecision.Order Decision Rule to the Rule Selection section on the Properties tab, as shown in Figure 12.
    Figure 12. Update the rules section
    Screen cap: Prompt for List the rules and rule packages in the rule task

Deploying the rules to the Rule Team Server

Once you have a complete rule project, publish the rule project to the Rule Team Server (RTS). With the rules in the RTS, it is possible for a business user to change the values you set. For example, if too many orders are considered exceptions, a business user could raise the threshold value from 500 to 1000.

To deploy to the RTS, complete these steps:

  1. Right-click the OrderDecisionRules project from the Rule explorer pane on the left, and click Rule Team Server > Connect.
  2. Enter your connection information.
  3. Ensure Create a new project on Rule Team Server is selected, and click Finish.

That's it! Business users can now log into the RTS and edit the rule you just published.

Publishing rules to the Rule Execution Server

In order to execute the rules, you have to publish the rules from the RTS to the Rule Execution Server (RES). The RES provides two capabilities:

  • It can execute the rules remotely and send the response back to the caller.
  • It can use the RES as a rule repository.

For the example scenario, you will connect to the RES and acquire the latest version of the rules, but you will execute them locally within your InfoSphere Streams environment for maximum performance.

Create a ruleapp

A ruleapp is a packaging format used to deploy rules on the RES. Once you've created a ruleset, it is packaged within a ruleapp and deployed onto the RES execution environment. Complete these steps to create a ruleapp:

  1. Log into the RTS, and click Configure > Manage RuleApps.
  2. Click New, and enter the name OrderDecisionRuleApp.
  3. Click New under Rulesets, and name it OrderDecisionRuleSet.
  4. Choose the OrderDecisionRules project, and click Save > Save to save the rule app.

Publish the ruleapp

Complete these steps to publish the ruleapp:

  1. Click Configure > Manage RuleApps.
  2. Check the box next to OrderDecisionRuleApp, and click Deploy.
  3. Un-check the box next to Create a baseline for this deployment, and click Next.
  4. Check Deploy on a Rule Execution Server, and click Next.
  5. Check Increment ruleset(s) major version, and click Next.
  6. Select your execution server from the list, and click Deploy.

Developing and integrating Streams applications

This section describes how to work with the Streams applications.

Application overview

The example Streams application provided in Downloads calculates a bargain index. Figure 13 shows a graph view of the application obtained from the StreamSight tool, which is included with InfoSphere Streams.

Figure 13. Streams application
Left to right flow of data, as described below

In Figure 13, data comes in from the source operator on the left and is then divided into two streams. One stream is for trades, and the other stream is for quotes. The trade stream is then aggregated and operated on to calculate the volume weighted average price (VWAP). The VWAP is then joined with the quote stream, and the bargain index is calculated. Finally, if the bargain index value is above a certain threshold, an order is generated.

For the example scenario, you will extend this application by placing an operator between the last two operators on the right side of the graph. This operator will call the JRules rule you already created to determine if the order can continue to be executed.

Extending the application

Complete these steps to extend the application by adding a JavaUdop operator.

  1. Extract the attached Streams application (orderDecision_orig.tar.gz) to a directory on your computer, such as a directory called orderDecision_orig, by entering tar -xvzf orderDecision_orig.tar.gz.
  2. Create a directory in the newly created orderDecision_orig directory called classpath, and copy the jars as shown.
    • In the ILOG studio lib directory:
      • asm-3.1.jar
      • asm-analysis-3.1.jar
      • asm-commons-3.1.jar
      • asm-tree-3.1.jar
      • asm-util-3.1.jar
      • bcel-5.1.jar
      • jdom-1.0.jar
      • jrules-engine.jar
      • sam.jar
    • In the db2 java directory:
      • db2jcc4.jar
    • In the ILOG executionserver lib directory:
      • jrules-res-execution.jar
      • j2ee_connector-1_5-fr.jar
  3. Configure the JRules connection by getting the ra.xml file in Downloads.
  4. Place the file in the classpath directory.
  5. Edit the file and enter the user name, password, and URL of your JRules RES db for the persistenceProperties property.
  6. Add the two operators shown in Listing 3 just before the sink operator at the end of the vwap.dps file to add the Java Udop to the Streams application. The vwap.dps file contains the Streams application shown in Figure 13.
Listing 3. Java UDOPs to split the order stream
	    stream OrdersNotToVerify(schemaFor(Orders))
		:= Functor(Orders)
		     [price * qty <= 200d]
		     {}	

	    stream OrdersToVerify(schemaFor(Orders))
		:= Functor(Orders)
		     [price * qty > 200d]
		     {}

These operators split the order stream into two separate streams. One stream contains orders that are small enough that you don't need to verify them. JRules needs to verify the other stream.

  1. Add the operators in Listing 4 after the OrdersToVerify operator to add the call to the JRules engine.
Listing 4. Updates to the OrdersToVerify operator
	    stream CheckedOrders(schemaFor(Orders), canContinue: Boolean)
	     := JavaUdop(OrdersToVerify)
	     [generated;
	     className:"com.ibm.orderchecker.OrderDecision";
	     classLibrary:
	     "../orderDecision_orig/classpath/asm-3.1.jar",
	     "../orderDecision_orig/classpath/asm-analysis-3.1.jar",
	     "../orderDecision_orig/classpath/asm-commons-3.1.jar",
	     "../orderDecision_orig/classpath/asm-tree-3.1.jar",
	     "../orderDecision_orig/classpath/asm-util-3.1.jar",
	     "../orderDecision_orig/classpath/bcel-5.1.jar",
	     "../orderDecision_orig/classpath/j2ee_connector-1_5-fr.jar",
	     "../orderDecision_orig/classpath/jdom-1.0.jar",
	     "../orderDecision_orig/classpath/jrules-engine.jar",
	     "../orderDecision_orig/classpath/jrules-res-execution.jar",
	     "../orderDecision_orig/classpath/sam.jar",
	     "../orderDecision_orig/classpath/db2jcc4.jar",
	     "../orderDecision_orig/classpath",
	     "../orderDecision_orig/bin";
 	     vmArg:"-Xmx512m"] {}
  1. Modify the paths for the classpath directory in the classLibrary parameter to the path you created above.
  2. After the call to JRules, you need to filter out all orders that did not pass the check and modify the Sink operator to print out the verified orders as well as the orders that weren't checked. Do this by replacing the Sink operator at the end of the file with the code in Listing 5.
Listing 5. Modify VerifiedOrders
	    stream VerifiedOrders(schemaFor(Orders))
	     := Functor(CheckedOrders)
	     [canContinue]
	     {}
	    
	    Null := 
	    Sink (OrdersNotToVerify, VerifiedOrders) ["file:///Orders.dat", nodelays]{}
  1. Save the vwap.dps file.
  2. Enter the make command from the orderDecision_orig directory to compile the application.
  3. When the compile completes successfully for the example scenario, you see the following:
    • A summary showing that 1 job and 12 processing elements (PEs) have been created.
    • The three operators that you created previously (OrdersToVerify.dpe, OrdersNotToVerify.dpe, and CheckedOrders.dpe).
    • A number of shell scripts.
    When you compile the Streams application for the first time with the JavaUdop operator in it, the application automatically creates the required .java and .class files under the src directory.
  4. Modify the OrderDecision.java file to call the ILOG JRules API by adding the bold lines from Listing 6 to the existing OrderDecision.java file. The comments in Listing 6 document the functionality being added. A modified OrderDecision.java file is included in Downloads if you need it.
Listing 6. Filter orders
	    package com.ibm.orderchecker;
	   
	    import ilog.rules.res.model.IlrPath;
	    import ilog.rules.res.session.IlrJ2SESessionFactory;
	    import ilog.rules.res.session.IlrSessionRequest;
	    import ilog.rules.res.session.IlrSessionResponse;
	    import ilog.rules.res.session.IlrStatelessSession;
	   
	    import com.ibm.streams.spade.OperatorContext;
	   
	    public  class OrderDecision extends AbstractOrderDecision {
	   
	   
	    	private IlrPath path;
	   	
	   	
	       /**
	       ** Initialize the operator
	       */
	       @Override
	       public void initialize(OperatorContext context) throws Exception {
	           super.initialize(context);
	   		
	   		//Set the path to the Rule. It is in the form /RuleApp/RuleSet
	   		this.path = 
	   		  IlrPath.parsePath("/OrderDecisionRuleApp/OrderDecisionRuleSet");
	       }
	   
	       /**
	       ** process method for port 0 (Stream IPort0Stream).
	       */
	       @Override
	       protected void process0(IPort0 tuple) throws Exception {
	   		//Initialize a new JRules Session. If you are concerned about 
	   		//performance, this session can be re-used. However if a 
	   		//new version of the rules are published to the RES, they 
	   		//will not be picked up until the session is re-initialized
	   		IlrJ2SESessionFactory sessionFactory = 
	   		  new IlrJ2SESessionFactory();
	           sessionFactory.setClassLoader(getClass().getClassLoader());
	   		
	   		
	   		//Create the JRules Request Object
	   		IlrSessionRequest sRequest = sessionFactory.createRequest();
	   		sRequest.setRulesetPath(path);	
	   		sRequest.setInputParameter("symbol", tuple.get_symbol());
	   		sRequest.setInputParameter("price", tuple.get_price());
	   		sRequest.setInputParameter("qty", tuple.get_qty());
	   
	   		IlrSessionResponse sResponse = null;
	   		try {
	   			//Execute the JRules Rule
	   			IlrStatelessSession ss = 
	   			  sessionFactory.createStatelessSession();
	   			sResponse = ss.execute(sRequest);
	   			
	   			//Parse the response
	   			boolean canContinue = 
	   ((Boolean)sResponse.getOutputParameters().get("canContinue")).booleanValue();
	   			
	   			
	   			//Create the Output Tuple. You have to set all of 
	   			//the input params as well as the JRules result
	   			OPort0 otuple = getOutput0().newTuple();
	   			
	   			otuple.set_symbol(tuple.get_symbol());
	   			otuple.set_price(tuple.get_price());
	   			otuple.set_qty(tuple.get_qty());
	   			
	   			otuple.set_canContinue(canContinue);
	   			
	   			//Submit the result tuple
	   			submit0(otuple);
	   			
	   		} catch (Exception e) {
	   			e.printStackTrace();
	   		}
	   		
	        }
	    }
  1. Enter the command ./start_streams_vwap.sh to start the Streams server.
  2. Enter the command ./submitjob_vwap.sh to submit the job.
  3. Use the StreamSight tool included with InfoSphere Streams to view the application by opening eclipse and selecting the InfoSphere Streams Live Graph perspective.
  4. Click the Load icon on the toolbar, and select Spade. Figure 14 shows the resulting application graph.
Figure 14. Modified Streams application
New data flow with an additional decision break in the path after the Orders function

In Figure 14, the data flows left to right in the same way it did in Figure 13, except when it reaches the Orders function, the flow splits into two paths before they converge again at the Sink function.

  1. Run the application. You modified the orderDecision.java file to invoke your ILOG rule that sets the canContinue flag to true if the price*quantity is less than $500 or if the symbol was IBM. After running the application, you'll see the output file, Orders.dat, in the orderDecision_orig/data directory. Within that file, you should only see orders that conform to your ILOG rule, as shown in Listing 7.
Listing 7. Generated Orders file
	    symbol:TWX,price:17.7,qty:1
	    symbol:UNH,price:64.6,qty:4
	    symbol:MRK,price:32.07,qty:3
	    symbol:VRX,price:18.67,qty:1
	    symbol:CD,price:16.75,qty:17
	    symbol:CVS,price:27.26,qty:10
	    symbol:CVS,price:27.26,qty:8
	    symbol:HAR,price:97.23,qty:1CH
	    symbol:IBM,price:83.59,qty:6
	    symbol:THC,price:7.93,qty:14
	    symbol:THC,price:7.93,qty:7
	    symbol:FSLb,price:26.04,qty:4
	    symbol:BHI,price:61.95,qty:3
	    symbol:LRY,price:43.67,qty:4
	    symbol:DG,price:19.18,qty:24
	    symbol:CEG,price:56.79,qty:1

Conclusion

This article showed the process of developing and deploying a business rule to a JRules application server and using this rule from an InfoSphere Streams application. The combination of these two technologies provides a powerful way for a business user to modify the rules used by a high performance data processing engine in real time without the need for development or deployment.


Download

DescriptionNameSize
Downloads for this articleStreamsJRulesCodeDownloads.zip831KB

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Information management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Information Management, WebSphere, Java technology, XML, Industries
ArticleID=502178
ArticleTitle=IBM technology in the financial markets front office, Part 2: Invoke conditional business rules based on analyzing data streams
publish-date=08052010