Example 2: Writing time series to dedicated OpenSearch indices based on the ruleset path

This example shows how to filter events to write time series to dedicated indices.

About this task

In this example, you customize the default Operational Decision Manager processing configuration to write the time series to dedicated OpenSearch indices, based on the ruleset path.

Such customization involves the following steps.

Step 1: Define a new OpenSearch egress that filters out the events and write the result to the new dedicated OpenSearch index.
Step 2: Update the existing OpenSearch index to exclude events that are based on the ruleset path or decisionServiceId.
Step 3: Declare a monitoring source to monitor data from the index with Business Performance Center.

In this example, you customize the default Operational Decision Manager processing configuration to write the time series to dedicated OpenSearch indices, based on the ruleset path. In the same way, you have an example of processing configuration for Automation Decision Services to write the time series to dedicated OpenSearch indices, based on the decisionServiceId.

Procedure

Define a new OpenSearch egress by duplicating the default OpenSearch egress and updating it.

Update the unique identifier (UID) by running the uuidgen command to generate a new identifier.
Update the index name to icp4ba-bai-odm-loanvalidation or icp4ba-bai-ads-loanvalidation, for example.

Add a selector to filter the events.

For Operational Decision Manager, based on the ruleset path, this selector is:

"selector": {
  "type": "jslt-inline",
  "expression" : "test(.source.id, \"^/loanvalidation/.*/loanvalidationrules/.*$\")"
}

This selector relies on standard regular expressions to match a specific ruleset path. Adjust the regular expression to match your needs, for example by reusing the following expressions.

To select events that match all versions of ruleset loanvalidationrules:
```
^/loanvalidation/1.0/loanvalidationrules/.*$
```
To select events that match versions of ruleset loanvalidationrules earlier that 2.0:
```
^/loanvalidation/1.0/loanvalidationrules/[0-1].[0-9]+$
```
To select events that match versions 2.0 and later of ruleset loanvalidationrules:
```
^/loanvalidation/1.0/loanvalidationrules/(?:[2-9]|[0-9]{2,}).[0-9]+$
```

The definition of the OpenSearch egress now looks like this.

{
  "uid": "<new generated uuid>",
  "type": "elasticsearch",
  "index": {
    "name": "icp4ba-bai-odm-loanvalidation",
    "docId": {
      "type": "jslt-inline",
      "expression": ".id"
    },
    "mapping": {
      "filename": "elasticsearch-mapping.json"
    }
  },
  "transformer": {
    "type": "jslt-file",
    "filename": "transformation.jslt"
  },
  "selector": {
    "type": "jslt-inline",
    "expression" : "test(.source.id, \"^/loanvalidation/.*/loanvalidationrules/.*$\")"
  }
}

For Automation Decision Services, based on the decisionServiceId value, this selector is:

"selector": {
  "type": "jslt-inline",
  "expression" : "test(.source.id, \"^LoanValidationDecisionService$\")"
}

The definition of the OpenSearch egress now looks like this:

{
 "uid": "<new generated uuid>",
 "type": "elasticsearch",
 "index": {
   "name": "icp4ba-bai-ads-loanvalidation",
   "docId": {
     "type": "jslt-inline",
     "expression": ".id"
   },
   "mapping": {
     "filename": "elasticsearch-mapping.json"
   }
 },
 "transformer": {
   "type": "jslt-file",
   "filename": "transformation.jslt"
 },
 "selector": {
   "type": "jslt-inline",
   "expression" : "test(.source.id, \"^LoanValidationDecisionService$\")"
  }
}

Update the existing OpenSearch.

Add a selector that filters out the other ruleset paths, by enclosing within the not() function the same selector expression as in step 1.

For Operational Decision Manager:

"selector": {
  "type": "jslt-inline",
  "expression" : "not(test(.source.id, \"^/loanvalidation/.*/loanvalidationrules/.*$\"))"
}

Your new egress now looks like this:

{
 "uid": "46D2CB58-2978-4DF5-909A-80583719A0A3",
 "type": "elasticsearch",
  "index": {
    "name": "icp4ba-bai-odm-timeseries",
    "docId": {
      "type": "jslt-inline",
      "expression": ".id"
    },
    "mapping": {
      "filename": "elasticsearch-mapping.json"
    }
  },
  "transformer": {
    "type": "jslt-file",
    "filename": "transformation-data.jslt"
   },
   "selector": {
    "type": "jslt-inline",
    "expression" : "not(test(.source.id, \"^/loanvalidation/.*/loanvalidationrules/.*$\"))"
  }
}

For Automation Decision Services:

"selector": {
  "type": "jslt-inline",
  "expression" : "not(test(.source.id, \"^LoanValidationDecisionService$\"))"
}

Your new egress now looks like this:

{  
  "uid": "46D2CB58-2978-4DF5-909A-80583719A0A3",
  "type": "elasticsearch",
  "index": {
    "name": "icp4ba-bai-ads-timeseries",
    "docId": {
      "type": "jslt-inline",
      "expression": ".id"
    },
    "mapping": {
      "filename": "elasticsearch-mapping.json"
    }
  },
  "transformer": {
    "type": "jslt-file",
    "filename": "transformation-data.jslt"
  },
   "selector": {
     "type": "jslt-inline",
    "expression" : "not(test(.source.id, \"^LoanValidationDecisionService$\"))"
  }
}

Create a dedicated monitoring source to monitor data from the new index.

For instructions, see Processing application.

For Operational Decision Manager:

- monitoring_source_type: 'Decisions (ODM) Loan Validation'
  elasticsearch_index: icp4ba-bai-odm-loanvalidation-ibm-bai
  monitoring_source_fields:
  - field: rulesetPath

For Automation Decision Services:

- monitoring_source_type: 'Decisions (ADS) Loan Validation'
  elasticsearch_index: icp4ba-bai-ads-loanvalidation-ibm-bai
  monitoring_source_fields:
  - field: decision.serviceId

What to do next

Before you apply the changes, you might be interested in further customizing your processing configuration, as explained in these examples.
- Example 3: Filtering out data
- Example 4: Obfuscating data
To apply your changes, follow instructions in Updating the processing configuration.
You can then send events to the Kafka ingress topic.