Introducing the Generic Ruleset Signature pattern for WebSphere Operational Decision Management V7.5, Part 3: Integrating Hadoop and IBM Operational Decision Management V8

The third part of this series describes how the Generic Ruleset Signature pattern can be used to integrate IBM® Operational Decision Management with Apache™ Hadoop™ to execute business rules within a distributed computing solution. This content is part of the IBM Business Process Management Journal.

Nigel T. Crowther (nigelcro@uk.ibm.com), Senior IT Consultant, IBM

Nigel Crowther photoNigel T. Crowther is a Senior IT Consultant in the IBM Software Services for WebSphere BPM Practice specializing in business rules. He has over 10 years experience in designing and developing business rules and business process management systems and 25 years experience in IT. Nigel has experience across all business sectors with his greatest focus on financial systems. Nigel joined IBM as part of the ILOG acquisition in 2009.



Jonathon Carr (jonathon.carr@uk.ibm.com), Staff IT Consultant, IBM

Jonathon Carr photoJonathon Carr is a Staff IT Consultant in the IBM Software Services for WebSphere BPM Practice specializing in business rules and visualization. Jonathon has over 10 years of experience in designing, building and documenting business rules and GUI applications. He joined IBM as part of the ILOG acquisition in 2009.



26 September 2012

Also available in Chinese

Introduction

This article presents a solution to integrating Apache Hadoop and IBM Operational Decision Management (IBM ODM) business rules. It uses the Generic Ruleset Signature (GRS) pattern described in Parts 1 and 2 of this series. First we'll present a generic integration pattern, then we'll show a trade validation example using the pattern. Finally we'll walk you through how to run the example.

What are Hadoop and IBM ODM?

Hadoop is an open source platform designed to solve problems with vast unstructured data. IBM ODM is a platform for business users to manipulate data. The combination of the two creates a powerful tool for business users to manipulate large volumes of data.

The core of Hadoop is the MapReduce framework, which processes data in parallel. This is achieved in conjunction with the Hadoop Distributed File System (HDFS), which manages the distribution of data. We'll discuss these components briefly, but for more detail refer to the Hadoop documentation.

Why integrate a business rules management system (BRMS)?

The conventional way to develop Hadoop applications is to build business logic in Java™ or with a scripting language like Pig or Hive. Introducing a BRMS enables the business logic to be externalized into rules and maintained in a business-friendly way. This has two key benefits:

  • The business can make changes.
  • It brings agility and governance to the change process.

Guidelines for ODM Hadoop integration

This section describes six guidelines for integrating ODM business rules with Hadoop:

  1. Call rules from the Map function

    Use the Map function to invoke the business rules. The Map function is where the meat of the data transformation takes place and is the most likely place for business rules. The Reduce function is primarily used for combining data, so it's a less useful place for business rules.

  2. Use a low-level rules engine

    Use the native rule engine API, IlrContext. This is the lowest level API; it is lightest in terms of size and works best with the Hadoop architecture. It allows Hadoop rather than the Rule Execution Server to control the distribution of the rules.

  3. Use flat data

    Use flat data such as CSV (comma-separated values) or tab delimited files. Flat data is compact, easily manipulated, and easily converted to a hash map where it can be ingested by the rule engine. Additional attributes can be added to flat data without having to change the rule set signature. See guideline 4.

  4. Use the Generic Ruleset Signature pattern for rule execution

    Use the Generic Ruleset Signature (GRS) pattern to allow for the addition and removal of data columns without resorting to code changes. For more information on the Generic Ruleset Signature pattern, refer to Parts 1 and 2 of this series.

  5. Batch the data

    Batching CSV data helps improve performance by reducing rule engine context switching. Instead of sending one row at a time to the engine, a batch is sent. For more details on setting the ideal batch size see Part 2.

  6. Build everything into a single JAR

    One of the key principles of Hadoop is that the executable is sent to the data, which it is faster than moving data to the executable. With this principle in mind the entire Hadoop job is packaged into a single JAR. The JAR file includes the MapReduce classes, the rules engine, the Hadoop core, and the business rules, as shown in Figure 1.

    Figure 1. Contents of the MapReduce job
    Contents of the MapReduce job

Run the Hadoop job

The task manager coordinates the MapReduce jobs. It ensures each Map job is given a part of the data known as the split. The Map sends the split to an instance of the rule engine where it is processed against the rules. The rules create interim data which is combined within the Reduce jobs, as shown in Figure 2.

Figure 2. Linking rule engine, Map and Reduce
Linking rule engine, Map and Reduce

A trade validation example

This section presents a trade validation example called Trade Validator, based on the guidelines described above. Trade validation is a good example of Hadoop and ODM integration because the data volume is high and the rules change frequently.

Trade data input

The input to the Trade Validator is a series of CSV split files, starting with file01.csv and ending with file04.csv. Figure 3 shows part of file01.csv. Note that the format of the CSV file correlates to the input parameters defined in Figure 6.

Figure 3. Input CSV file format
Input CSV file format

In Figure 3 there is an erroneous trade at line 206 where the transaction attribute is XXX. This does not conform to the rule that states a transaction should be either BUY or SELL. This will produce a validation error. Validation errors are appended to the output file called validationErrors.csv. A validation error contains the split filename, the line number, the customer ID, the attribute that failed and a reason. Figure 4 shows the correlation between the input and output.

Figure 4. Validation errors correlated to input split
Validation errors correlated to input split

Top-level rule project

The top-level rule project is called grs-ruleflow-batch and is shown in Figure 5. Its input and output parameters are a batch of hash maps. Each transaction in the batch is processed until all there are no more transactions.

Figure 5. Top-level rule project
Top-level rule project

Input parameters

The input parameters into the rule engine are defined in Figure 6. They are defined in four GRS parameter tables, one for each primitive type, Integer, String, Double and Date.

Figure 6. Input variables
Input variables

Output parameters

The rule engine output contains one or more ErrorStructures. An ErrorStructure holds the error code, error message and the name of the parameter, as shown in Figure 7.

Figure 7. Error structure
Error structure

Validation rules

Listing 1 shows the validation business rules.

Listing 1. Trade validation rules
//Check Transaction Date
if
    the date transactionDate is after today
then
    add error for transactionDate reason "Date is in future" ;
    
//Check Total
definitions
    set price to the decimal stockPrice ;
    set volume to the integer units ;
    set checkTotal to price * volume ;
if
    the decimal total is not checkTotal
then
    add error for total reason "Total is not correct.  Expected " + checkTotal + 
    " but was " + the decimal total;    
        
//Check Transaction Type
if
    the string transaction is not one of { BUY , SELL }
then
    add error for transaction reason "Invalid transaction" ;
    
//Check Stock Name
if
    the string stockName is not one of { IBM , XYZ }
then
    add error for stockName reason "Invalid stockname" ;

The MapReduce code

This section describes the Java code for the MapReduce jobs and the integration layer to the rule engine.

The ValidateTrades class

The run method of the ValidateTrades class in Listing 2 is invoked at the start of the Hadoop job. At line 12 the job is supplied configuration parameters and the name of the class. At line 18 the mapper is set to the TradeMap class described later. Lines 19 and 20 are commented out so that the default combiner and reducer are used. At lines 25 and 26 the input and output directories are set. Finally at line 28 the job is run.

Listing 2. ValidateTrades class
09 public class ValidateTrades extends Configured implements Tool {
10
11        public int run(String[] args) throws Exception {
12                JobConf conf = new JobConf(getConf(), ValidateTrades.class);
13                conf.setJobName("TradeValidator");
14
15                conf.setOutputKeyClass(Text.class);
16                conf.setOutputValueClass(Text.class);
17
18                conf.setMapperClass(TradeMap.class);
19                // conf.setCombinerClass(Reduce.class);
20                // conf.setReducerClass(Reduce.class);
21
22                conf.setInputFormat(TextInputFormat.class);
23                conf.setOutputFormat(TextOutputFormat.class);
24
25                FileInputFormat.setInputPaths(conf, new Path(args[0]));
26                FileOutputFormat.setOutputPath(conf, new Path(args[1]));
27
28                JobClient.runJob(conf);
29
30                return 0;
31        }
32
33        public static void main(String[] args) throws Exception {
34                int res = ToolRunner.run(new Configuration(), new ValidateTrades(),
35                                args);
36                System.exit(res);
37        }
38 }

The TradeMap class

The TradeMap class in Listing 3 contains the Hadoop map method to invoke the rules.

At line 17 the RuleEngineRunner is created. It is initialized at line 27 within a static initializer, so that it is created once and only once at the start of the map job. The map method at line 31 accepts a single line of CSV data. It converts the data into a map of key/values pairs at line 39. The map is added into the batch at line 40. If the batch is full or you are at the end of the file (line 43), the batch is sent to the rule engine for processing (line 45). Batching helps improve performance of the rule engine by reducing the number of invocations. The result is a CSV string containing validation errors (line 45). If the number of errors is non-zero (line 47) then the errors are added to the output collector (line 50). The output collector collates the errors, which are eventually written to the output file. At line 54 the current status is sent to the reporter.

Listing 3. TradeMap class
14 public class TradeMap extends MapReduceBase implements
15 Mapper<LongWritable, Text, Text, Text> {
16
17      private static RuleEngineRunner runner = new RuleEngineRunner();
18      static final String EOF = "EOF";
19      private Text id = new Text();
20      private Text errors = new Text();
21      private long numRecords = 0;
22      private long numValidationErrors = 0;
23      private String inputFile;
24      private Batch batch = new Batch();
25
26      static {
27              runner.initialise();
28      }
29
30      @Override
31      public void map(LongWritable key, Text value,
32                      OutputCollector<Text, Text> output, Reporter reporter)
33                      throws IOException {
34
35              String csvLine = value.toString();
36
37              if (!csvLine.startsWith(EOF)) {
38
39                      Map<String, String> parameters = MapFactory.csvToMap(csvLine);
40                      batch.addItem(parameters);
41              }
42
43              if (batch.isFull(++numRecords) || csvLine.startsWith(EOF)) {
44
45                      String result = runner.runValidationRules(batch.getBatch());
46
47                      if (result.length() > 0) {
48                              errors.set(result);
49                              id.set(inputFile);
50                              output.collect(id, errors);
51                              ++numValidationErrors;
52                      }
53
54                      reporter.setStatus("Processed " + numRecords + " records from "
55                                      + inputFile + " with " + numValidationErrors
56                                      + " validation errors");
57
58                      batch.clear();
59              }
60      }
61
62      public void configure(JobConf job) {
63
64              inputFile = job.get("map.input.file");
65
66              int fileNamePos = inputFile.lastIndexOf('/') + 1;
67              inputFile = inputFile.substring(fileNamePos);
68
69              String colsStr = job.get("column.names", "");
70              MapFactory.getColumns(colsStr);
71
72      }

The MapFactory class

The MapFactory class in Listing 4 is a static factory that builds Java hash maps from CSV data. The getColumns method at line 18 tokenizes the comma-separated list of column names. These names are taken from the command line and used to populate the keys in the hashmap.

The csvToMap method at line 39 tokenizes the comma-separated list of CSV data. This data is used to populate the values in the hash map.

Listing 4. MapFactory class
07 public class MapFactory {
08
09      int MAX_COLS = 20;
10      private static String[] cols = new String[MAX_COLS];
11
12      /**
13       *
14       * @colsStr get the column names from the input parameters and 
15       *          copy it into an internal structure
16       * @return void
17       */
18      public static void getColumns(String colsStr) {
19
20              int i = 0;
21              StringTokenizer itr = new StringTokenizer(colsStr, ",");
22
23              // Get Column Names. This will be the key values in the 
24              // parameter hash map
25              while (itr.hasMoreTokens() && i < MAX_COLS) {
26                      cols[i] = itr.nextToken();
27                      System.out.println("Col[" + i + "]=" + cols[i]);
28                      i++;
29              }
30      }
31
32      /**
33       *
34       * @param row
35       *            of CSV data representing a set of parameters to be
36       *            passed to the rule engine
37       * @return parameters as a HashMap
38       */
39      public static Map<String, String> csvToMap(String row) {
40
41              // Get parameters which are defined on a row of CSV data.
42              StringTokenizer itr = new StringTokenizer(row, ",");
43
44              Map<String, String> args = new HashMap<String, String>();
45              int i = 0;
46
47              while (itr.hasMoreTokens() && i < MAX_COLS) {
48
49                      String valueStr = itr.nextToken();
50
51                      // cols is the key containing the parameter name
52                      args.put(cols[i++], valueStr);
53              }
54
55              return args;
56      }
57 }

The Batch class in Listing 5 encapsulates a data batch to be processed by the rule engine. At line 7 the constant BATCH_SIZE is set. This controls the number of rows in one batch. The constant should be set so that the batch easily fits in to physical memory within the JVM. At line 8 the batch is created as a list of maps. Each map contains one row of CSV data. At line 17 the method addItem adds a single row to the batch. At line 31 the method isFull determines whether the batch is full and ready for processing.

Listing 5. Batch class
05 public class Batch {
06
07 private static int BATCH_SIZE = 100;
08      private java.util.List<Map<String, String>> batch = 
09                               new java.util.ArrayList<Map<String, String>>();
10
11      /**
12       * Add a CSV row to the batch
13       * 
14       * @param parameters
15       *            The parameters to add to the batch
16       */
17      public void addItem(Map<String, String> parameters) {
18              batch.add(parameters);
19      }
20
21      /**
22       * Returns <code>true</code> if the batch is the right size for processing
23       * and <code>false</code> otherwise.
24       * 
25       * @param numberOfRecords
26       *            The number of records currently in the batch
27       * @return <code>true</code> if the batch is full and ready for processing
28       *         and <code>false</code> otherwise.
29       * 
30       */
31      public boolean isFull(long numberOfRecords) {
32
33              return ((numberOfRecords % BATCH_SIZE) == 0) ? true : false;
34      }
35
36      /**
37       * Clear the batch.
38       */
39      public void clear() {
40              batch.clear();
41      }
42
43      /**
44       * @return the real batch
45       */
46      public java.util.List<Map<String, String>> getBatch() {
47              return batch;
48      }
49 }

The RuleEngineRunner class

The RuleEngineRunner is a lightweight wrapper over the rule engine API. Following are the key methods within this class.

Initialise method

In Listing 6 the initialise method reads the rules in validationRules.jar (line 146), which is referenced using a relative path because the validationRules.jar is embedded within the Hadoop job JAR. The rule set is loaded into the rules engine (line 150) and then validated and built (line 153). After executing these two lines, the rule engine is ready for invocation.

Listing 6. Initialise method of RuleEngineRunner
143        public void initialise() {
144
145             // The relative path to the validation rules 
146             final String rulesetPath = "../validationRules.jar";
147
148             try {
149                     // Load the rule set
150                     loadRuleset(rulesetPath);
151
152                     // Prepare an engine for this rule set
153                     buildEngine();
154
155             } catch (Exception exception) {
156                     exception.printStackTrace();
157             }
158     }

LoadRuleset method

In Listing 7, the loadRuleset method uses the low-level engine API to load and parse the validation rules. Lines 43-47 create a stream to read the rule set. Line 49 creates an empty IlrRuleset. Line 53 initializes the ruleset parser. Line 55 parses the rule set and lines 60-68 handle errors.

Listing 7. LoadRuleset method of RuleEngineRunner
42      public void loadRuleset(String rulesetPath) throws Exception {
43
44              JarInputStream is = new JarInputStream(
45                       RuleEngineRunner.class.getResourceAsStream(rulesetPath));
46
47              IlrRulesetArchiveLoader rulesetloader = new IlrJarArchiveLoader(is);
48              IlrRulesetArchiveParser rulesetparser = new IlrRulesetArchiveParser();
49
50              ruleset = new IlrRuleset();
51
52              System.out.println("*****Loading Ruleset: " + rulesetPath);
53
54              rulesetparser.setRuleset(ruleset);
55
56              if (rulesetparser.parseArchive(rulesetloader)) {
57                      return;
58              }
59
60              String[] errors = rulesetparser.getErrors();
61
62              if (errors != null) {
63                      StringBuffer buf = new StringBuffer();
64                      for (int i = 0; i < errors.length; i++) {
65                              buf.append(errors[i]).append('\n');
66                      }
67                      System.out.println("*** ERROR Loading Ruleset: " + ruleset);
68                      throw new Exception(buf.toString());
69              }
70      }

BuildEngine method

In Listing 8, the buildEngine method uses the low-level engine API to create an instance of the rule engine. At line 76 an IlrContext is created using the validation rules. Lines 78-87 set up the error handlers.

Listing 8. BuildEngine method of RuleEngineRunner
75      public void buildEngine() throws IlrBadContextException {
76              engine = new IlrContext(ruleset);
77
78              IlrExceptionHandler exceptionHandler = new IlrExceptionHandler() {
79                      public boolean handleException(IlrUserRuntimeException ex) {
80                              if (ex.isInConditions())
81                                      return false;
82                              else
83                                      throw ex;
84                      }
85              };
86              engine.setExceptionHandler(exceptionHandler);
87              engine.setRuleflowExceptionHandler(exceptionHandler);
88      }

RunValidationRules method

In Listing 9, the runValidationRules method sets the rule engine parameter to the batch (lines 140-146). The rules are executed against the batch (line 147). The output from the rule engine is a list of maps (line 150). Each map contains one or more validation errors against a trade. The output is appended to the validationResponse (line 158-159) which contains all errors for the batch.

Listing 9. RunValidationRules method of RuleEngineRunner
134     @SuppressWarnings("finally")
135     public String runValidationRules(List<Map<String, String>&t; batch) {
136
137             StringBuffer validationResponse = new StringBuffer();
138             try {
139
140                     IlrParameterMap inputs = new IlrParameterMap();
141                     
142                     // Set the request Parameter
143                     inputs.setParameter("request", batch);                  
144
145                     // Execute the rules
146                     engine.setParameters(inputs);
147                     IlrParameterMap outputs = engine.execute();
148
149                     // Retrieve the validation errors
150                     java.util.List<?> errors =  
151                           (java.util.List<?>) outputs.getObjectValue("errorList");
152
153                     for (Iterator<Map> errorsIter = 
154                        (Iterator<Map>)errors.iterator();errorsIter.hasNext();) {
155                             Map errorsMap = (Map) errorsIter.next();
156
157                             // iterate over values only
158                             for (Object error : errorsMap.values()) {
159                                     validationResponse.append(", " + error);
160                             }
161
162                             validationResponse.append("\n");
163
164                     }
165
166             } catch (Exception exception) {
167     }       exception.printStackTrace();
168             } finally {
169                     // Reset is necessary to execute the engine with a new batch
170                     reset();
171                     return validationResponse.toString();
172             }
173     }
174 };

Invoke the Hadoop job

You can initiate the Trade Validator job from the command line using the syntax shown in Figure 8.

Figure 8. Invoke the Trade Validator Hadoop job
Invoke the Trade Validator Hadoop job

Note: This command is provided as a text file in the download for this article.

Running status and completion

When the Hadoop jobs are running, you can see the status of each task in the Hadoop console. The status screen in Figure 9 shows the information for four map tasks: three have completed and one is 90% complete.

Figure 9. Viewing status of the Trade Validator Hadoop job
Viewing status of the Trade Validator Hadoop job

Run the Hadoop Trade Validator on Ubuntu

In this section, we'll walk through the steps to run the Hadoop Trade Validator on Ubuntu. First you'll run the Hadoop Trade Validator with the rules and data provided, then you'll modify the rules and data and rerun it.

Getting started

Before you get started, complete the following steps to prepare your environment.

  1. Download and install Ubuntu.
  2. Download hadoop-1.0.3-bin.tar.gz.
  3. Install IBM Operational Decision Management V8.
  4. Create a directory called Hadoop in /home/[username].
  5. Extract the Hadoop TAR into /home/[username]/Hadoop. This creates a hadoop-1.0.3 directory.
  6. Download tradeValidator.tar.gz provided with this article, and copy it to /home/[username]/Hadoop. This should create two directories: tradeValidator and tradeValidatorRules.
  7. In the tradeValidator directory, edit the build.properties. Set wodm.home.dir to point to your IBM Operational Decision Management 8.0 installation directory; for example:

    wodm.home.dir=/home/myusername/IBM/WODM80

    Also set hadoop.lib.dir and hadoop.core to point to your Hadoop installation. This should be correct if you have followed the previous steps; for example:


    hadoop.lib.dir=../hadoop-1.0.3
    hadoop.core=hadoop-core-1.0.3.jar
  8. Install Apache Ant with the command: sudo apt-get install ant1.7
  9. Now install the JDK with the command: sudo apt-get install openjdk-7-jdk
  10. In the /home/[username]/Hadoop/tradeValidator directory, build the code by typing: ant

    You should see the message BUILD SUCCESSFUL.

  11. Edit hadoop-env.sh in the hadoop-1.0.3/conf directory to ensure that JAVA_HOME matches your installation; for example:
    export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-i386

Run the Hadoop Trade Validator

In the /home/[username]/Hadoop/tradeValidator directory, give the run.sh executable permissions and then run the Hadoop job in standalone mode by issuing the following command:

chmod +x run.sh
./run.sh

The results are written to the output folder and to the screen. The expected validation errors are shown in Figure 10.

Figure 10. Output from the Hadoop job
Output from the Hadoop job

Modify the Trade Validator rules and input data

In this section, we'll modify some of the rules and input data and rerun the Trade Validator.

Import the rules projects

To import the rules projects, do the following:

  1. Import the folder tradeValidatorRules into Rule Designer. You should see the projects shown in Figure 11.
    Figure 11. Trade Validator projects
    Trade Validator projects
  2. Click Finish. All the projects should build without errors.

Add a new input parameter

To add a new input parameter, do the following:>

  1. Edit the Domains.xlsx file in the tradeValidatorRules/grs-bom/resources directory and add a new domain item to the String worksheet called currency, as shown in Figure 12.
    Figure 12. New currency input parameter
    New currency input parameter
  2. In the same spreadsheet, insert a new worksheet called CurrencyName with the contents in Figure 13. To speed things up, you can copy the contents of the StockName sheet, changing the value IBM to USD and the value XYZ to GBP.
    Figure 13. Currency names
    Currency names
  3. Save and close the Domains.xlsx file.
  4. Back in Rule Designer, right-click tradeValidatorRules/grs-bom/resources and refresh the folder.
  5. Open the StringParameter class in bom/domains/dynamic/dt/parameters of the grs-bom rule project.
  6. Click the Synchronize link to pick up the new currency domain item. Figure 14 shows the StringParameter class with the new currency domain value. Click Save.
    Figure 14. Updated StringParameter domain
    Updated StringParameter domain
  7. In the StringDomains package, add a new class called CurrencyNameDomain. Make sure the class has a default verbalization and has java.lang.String as the Execution name and the Superclass.
  8. Click Create a domain and select Excel Dynamic Domains.
  9. Select the Domains.xlsx file and create the values as shown in Figure 15.
    Figure 15. Create the CurrencyName domain
    Create the CurrencyName domain
  10. Click Finish and then Save. The final class definition is shown in Figure 16.
    Figure 16. CurrencyNameDomain class
    CurrencyNameDomain class

    (See a larger version of Figure 16.)

  11. Configure the new currency input parameter by adding a row to the rules/parameters/StringParameters decision table in the grs-bom project, as shown in Figure 17.
    Figure 17. Configure the currency string parameter
    Configure the currency string parameter
  12. Add a new action rule called CheckCurrency to the grs-rules rule project in the Validation package as shown in Listing 10.
Listing 10. CheckCurrency rule
if
    the string currency is not one of { GBP , USD }
then
    add error for currency reason "Invalid Currency"
  1. Regenerate the ruleset archive by right-clicking on the grs-ruleflow-batch rule project and selecting Export => Ruleset archive. Specify validationRules.jar as the archive file and save it in /home/[username]/Hadoop/tradeValidator/resources.
  2. To modify the input data, in the tradeValidator/input directory, add USD at the end of every row of the CSV files using the command: sed -i 's/$/,USD/' *.csv
  3. Edit file01.csv, and modify the second to last line so that the currency value is AUD instead of USD. Note: Do not alter the very last line, which is the End of File (EOF) indicator.

Rerun the Hadoop Trade Validator

In /home/[username]/Hadoop/tradeValidator, rebuild the code from the command line by typing: ant

You should see the message BUILD SUCCESSFUL. This build will have incorporated the latest validation rules into the Hadoop JAR file.

In the same directory, edit the run.sh file and append currency to the column.names parameter as follows:

hadoop jar tradevalidator.jar
tradevalidator.ValidateTrades
-Dcolumn.names=lineNumber,customerId,customerName,stockName,
transaction,transactionDate,units,stockPrice,total,currency 
input output

From the same directory, rerun the Hadoop job for the Trade Validator by typing: ./run.sh

The results are written to the output folder and to the screen. The expected validation errors should be as shown in Figure 18. Note that the invalid currency AUD has been detected in file01.csv.

Figure 18. Output from rerun Hadoop job
Output from rerun Hadoop job

Conclusion

This concludes our three-part series on the GRS pattern for business rules. Part 1 provided a simple overview of the pattern and its advantages and disadvantages. In Part 2 we battle-hardened the original implementation of GRS for real-world use. In this final article, we stepped through a real example showing how GRS can be used with Hadoop for high-volume trade validation.


Acknowledgements

The authors would like to thank Pierre Berlandier and Richard G. Brown for reviewing this paper and special thanks to Scott Gray for his expertise on implementing Hadoop solutions.


Download

DescriptionNameSize
Hadoop trade validator with business rulestradeValidator.zip78KB

Resources

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Business process management on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Business process management, WebSphere
ArticleID=837500
ArticleTitle=Introducing the Generic Ruleset Signature pattern for WebSphere Operational Decision Management V7.5, Part 3: Integrating Hadoop and IBM Operational Decision Management V8
publish-date=09262012