KPI rule language
KPI rules contain first-level nodes of name and instructions. The full rule language is map, filter, query, dimension, window, sendEventsWhenNoData, and reduce.
Example rule
{
"name":"kpi-docVolume-kpi-rule",
"instructions":{
"dimension":[
{
"fieldPath":"eventDetails.transactionType",
"type":"String"
}
],
"window":{
"timeWindows":[
"day"
],
"fieldPath":"timestampEventOccurred"
},
"sendEventsWhenNoData":"yes",
"reduce":{
"expression":{
"operator":"sum",
"fieldPath":"eventDetails.docVolume",
"type":"long"
},
"output":{
"resultAlias":"documentVolume",
"resultType":"long"
}
}
}
}Rule structure overview
{
"name":"my KPI rule",
"instructions":{ // required
"query": [ // optional
{
"basedObjectType": "",
"select": [ "field1", "field2", "field3", ... ]
"from" : "",
"where" : {
"equals": [
{
"fieldPath": ["field2","string"]
},
{
"valueFrom": ["field1","string"]
}
]
}
}
],
"map":[ // optional
{
"output":{
"resultFieldPath":"fieldx",
"resultType":"int"
},
"expression":{
"DIVIDE":[
{
"MULTIPLY":[
{
"SELECT":["field1","int"]
},
{
"VALUE":[2,"int"]
}
]
},
{
"VALUE":[2,"int"]
}
]
}
}
],
"filter":{ //optional
"OR":[
{
"AND":[
{
"EQUALS":[
{
"SELECT":["field1","int"]
},
{
"SELECT":["field2","int"]
}
]
},
{
"GREATERTHAN":[
{
"SELECT":["field3","int"]
},
{
"VALUE":[100,"int"]
}
]
}
]
},
{
"EQUALS":[
{
"SELECT":["field4","int"]
},
{
"SELECT":["field5","int"]
}
]
}
]
},
"dimension":[ // optional
{
"fieldPath":"", // all fields about "fieldPath" cannot contain "-".
"type":""
}
],
"window":{ // required
"timeWindows":["HOUR"],
"fieldPath":"timestampEventOccurred"
},
"sendEventsWhenNoData":"yes", // optional
"reduce":{ // required
"expression":{
"operator":"sum",
"fieldPath":"",
"type":""
},
"output":{
"resultType":"",
"resultAlias":""
}
}
}
}
KPI rules contain first-level nodes of name and instructions.
Both nodes are required.
| Property | Description |
|---|---|
| Name | The name of the KPI rule. |
| Instructions | Operations on the data. |
Instructions consist of map, filter, query,
dimension, window, sendEventsWhenNoData, and
reduce.
Map (Optional)
A map is a list of operations that define how to calculate the specified field with given expression. The following example shows the scope:
"map": [
// the first
{
// the output field and type
"output": {
"resultFieldPath": "eventDetails.totalValue",
"resultType": "double"
},
// how to calculate the output
"expression": {
"multiply": [
{
"select": [
"eventDetails.productLines.unitPrice", // field
"double" // type
]
},
{
"select": [
"eventDetails.productLines.quantity",
"double"
]
}
]
}
},
// the second
{
...
}
]Every map consists of required output and an optional
expression.
| Root level | Second level | Description |
|---|---|---|
| Output | resultFieldPath |
The field of the calculated result. This value between elements in map is unique. |
resultType |
The type of the calculated result in one of the following values [int, long, double, float]. | |
| Expression | Nested | The definition for how to calculate the output. |
The expression supports the multilevel nest. The following example shows
a valid expression:
{
operator:[ // root level
{
// second level
operator: [ operand, operand ]
},
{
// second level
operator: [ operand, operand ]
}
]
}The operator in the root level of the expression must be one of the
following types:
MULTIPLY, DIVIDE, ADD, SUBTRACT, MOD, MIN, MAX, ABS, SQRT,
ROUNDThe number of operand depends on the concrete operator in the
preceding list. For example, MULTIPLY must have two operands, such as a * b, while
ABS only needs one operand. Using the wrong operand number creates an invalid KPI
rule.
The operator in the second level can support the following types:
MULTIPLY,
DIVIDE, ADD, SUBTRACT, MOD, MIN, MAX, ABS, SQRT, ROUND, SELECT, VALUEIf you use an extra
operator, you can use SELECT and VALUE as the end-operator, which
can be defined to mark the end of nesting of map, for
example,
{
"MULTIPLY":[
{
"SELECT":[ "field1", "int"]
},
{
"VALUE":[ 2, "int"]
}
]
}The root-level operator is MULTIPLY, the first operand is
"SELECT field1", which the operator is SELECT and the second
operand is "set value of 2". So this part means "field1 *
2".
In the structure of end-operand, the operand is an array whose
size must be 2. The first object is the field name of the event schema or the value of a number,
such as "field1" or 2 in the previous example. The second object
is the type of the value, which should be in the list of [int, long, double, float].
Filter (Optional)
The filter
is a nested JSONObject that is used for filtering data with a given expression. It
uses the following structure:
"filter": {
// root-level operator
"or": [
{
// second-level operator, first operand of root-level
"greaterthan": [
{
"select": ["eventDetails.totalValue","double"]
},
{
"value": [300, "double"]
}
]
},
{
// second-level operator, second operand of root-level
"equals": [
{
"select": ["eventDetails.owningOrganization", "string"]
},
{
"value": ["ibm", "string"]
}
}
]
}It operates like the expression. The root level operator must be
in:
GREATER_THAN, GREATER_EQUALS, LESS_THAN, LESS_EQUALS, EQUALS,UNEQUALS, OR,
ANDAnd the other level operator can match SELECT and
VALUE. It is almost the same as an expression.
When the operator matches
EQUALS, there must be one SELECT operand, for
example,
// The following code snippet is not in the expected format
{
"EQUALS":[
{
"VALUE":[2,"int"]
},
{
"VALUE":[2,"int"]
}
]
}
// The following code snippet is in the expected format
{
"EQUALS":[
{
"SELECT":["field","int"]
},
{
"VALUE":[2,"int"]
}
]
}
// The following code snippet is in the expected format
{
"EQUALS":[
{
"SELECT":["field","int"]
},
{
"SELECT":["field2","int"]
}
]
}Query (Optional)
In a KPI business rule,query implements the ability to use the result of that query as input into the KPI
processing, in addition to the input events. For example, when you want a KPI for order lines, you
probably also want information from the order. You can use query to select the field you
want.{
"query":[
{
"basedObjectType":"order",
"select":[
"sourceId",
"eventDetails.order",
"eventDetails.product",
"eventDetails.quantity",
"eventDetails.productValue",
"eventDetails.value"
],
"from":"orderLine",
"where":{
"and":[
{
"equals":[
{
"fieldPath":[
"eventDetails.order", // based event
"string"
]
},
{
"valueFrom":[
"eventDetails.orderId", // from event
"string"
]
}
]
},
{
"greaterthan":[
{
"fieldPath":[
"eventDetails.quantity",// based event
"integer"
]
},
{
"value":[
5,
"integer"
]
}
]
}
]
}
}
]
}Query structure
A query consists of the following required elements:
| Root level | Description |
|---|---|
basedObjectType |
The based event that needs its field as condition to run the query. |
select |
The fields that you want to query. |
from |
The event to be queried. |
where |
Query conditions, its usage is the same as filter
|
fieldPath means that its value comes from the target object, and there
are two ways to get the condition value:-
valueFrom, the condition value comes from thebasedObjectTypeevent. -
value, the condition value is obtained directly.
For example, the where condition means that there two query conditions. The
first is that the eventDetails.order in order event should be
equal to the value of eventDetails.orderId in orderLine event, and
the second condition is that the eventDetails.quantity should be greater than
5.
The other usage of the where condition is same as the filter expression.
Dimension (Optional)
"dimension": [
{
"fieldPath": "tenantId",
"type": "String"
},
{
"fieldPath": "sourceId",
"type": "String"
},
{
"fieldPath": "objectType",
"type": "String"
}
]It is an array, where every element object contains a fieldName and a type, and
both properties are required.
| Property | Description |
|---|---|
fieldPath |
The name of the dimension field in the event. |
type |
The type of value for the field in the conditionFields object
element. |
Window (Required)
"window": {
"timeWindows": [
"day",
"month",
"year"
],
"fieldPath": "timestampEventOccurred"
}It mainly contains timeWindows and fieldPath. Both properties
are required.
| Property | Description |
|---|---|
timeWindows |
An array to describe aggregation granularity. |
fieldPath |
The name of the dimension field in the event schema |
The value of TimeWindows must be in the list of:
SECOND, MINUTE, HOUR, DAY, MONTH, YEARThese values can be existent at the same time and split by using a comma (,),
such as day,month.
SendEventsWhenNoData (Optional)
ThesendEventsWhenNoData Boolean field governs whether to fire a window-event at the
end of the window period none of the ingest data matches the rule within that window.The default value for this field is false, and the lack of an event within that window means that the value is '0' for all dimensions.
The value of sendEventsWhenNoData must be in the list
of:
yes, noReduce (Required)
"reduce": {
"expression": {
"operator": "sum",
"fieldPath": "eventDetails.totalValue",
"type": "double"
},
"output": {
"resultAlias": "totalValue",
"resultType": "double"
}
}The following properties are required for reduce:
| Property | Nest | Description |
|---|---|---|
| expression | operator |
The math operation |
fieldPath |
The name of field that will be calculated in event | |
type |
The type of value for 'field' | |
| output | resultAlias |
The alias of calculated result |
resultType |
The type of calculated result |
The operator must be in list of "COUNT", "COUNT_DISTINCT",
"SUM".
The type must be in list of int, long, double, float. Also, the
resultType must be the same as the type.
Data flow of event to ingest service
- Original event
- The KPI service retrieves event data from Kafka and then aggregates data from the dimension of
time. The following example shows original data, which is produced in the upstream
service:
{ "eventCode":"objectUpsertEvent", "objectType":"com.ibm.infohub.businessEventModel.flowsummary", "infoHubObjectId":"15cfd5a6-9819-11e7-bc43-526af7764f69", "sourceId":"com.ibm.infoHub.kpiService", "tenantId":"3fed53d0-9817-11e9-bc42-526af7764f64", "timestampEventOccurred":"2016-01-01T13:40:16+0000", "timestampEventReceived":"2016-06-21T13:40:16+0000", "eventDetails":{ "businessObject":{ "id":"filter-mathOpt_1", "type":"Order", "seller":{ "name":"filter-mathOpt-seller0" }, "buyer":{ "name":"LENOVO_1" }, "owningOrganization":{ "name":"1111:RETAIL" }, "billToOrganization":{ "name":"AIG-9" }, "orderReferenceId":"100013204", "lineCount":2, "createdDate":"2019-09-27T13:40:16+0000", "requestedShipDate":"2019-09-27T13:40:16+0000", "requestedDeliveryDate":"2019-09-27T13:40:16+0000", "plannedShipDate":"2019-09-27T13:40:16+0000", "plannedDeliveryDate":"2019-09-27T13:40:16+0000", "lastModifiedDate":"2019-09-27T13:40:16+0000", "quantity":-6, "quantityUnits":"ea", "unitPrice":20.19, "unitPriceCurrency":"USD", "orderValue":-120.14, "orderValueCurrency":"USD", "totalAmount":-91.46, "totalAmountCurrency":"USD", "orderLine":[ { "id":"ABCD-1", "order":{ "id":"filter-mathOpt_1" }, "product":{ "description":"HMC CR1" }, "quantity":1, "quantityUnits":"ea", "productValue":111.1, "value":110, "valueCurrency":"USD", "orderLineNumber":"100", "shipmentCount":1, "createdDate":"2019-09-27T13:40:16+0000", "requestedShipDate":"2019-09-27T13:40:16+0000", "requestedDeliveryDate":"2019-09-27T13:40:16+0000", "plannedShipDate":"2019-09-27T13:40:16+0000", "plannedDeliveryDate":"2019-09-27T13:40:16+0000", "lastModifiedDate":"2019-09-27T13:40:16+0000" }, { "id":"ABCD-2", "order":{ "id":"filter-mathOpt_1" }, "product":{ "description":"HMC CR2" }, "quantity":5, "quantityUnits":"ea", "productValue":20.1, "value":100, "valueCurrency":"USD", "orderLineNumber":"100", "shipmentCount":5, "createdDate":"2019-09-27T13:40:16+0000", "requestedShipDate":"2019-09-27T13:40:16+0000", "requestedDeliveryDate":"2019-09-27T13:40:16+0000", "plannedShipDate":"2019-09-27T13:40:16+0000", "plannedDeliveryDate":"2019-09-27T13:40:16+0000", "lastModifiedDate":"2019-09-27T13:40:16+0000" } ] } } }
- Delta event
- The KPI service makes a preliminary aggregation under the KPI rule. If the preceding KPI rule
defines the time window of
day, month, year, then the KPI service generates three level events that correspond to the three time levels. The following example shows a simple delta event with day-level:{ "eventCode":"businessEvent", "objectType":"com.ibm.infohub.kpi.service.deltaEvent", "sourceId":"com.ibm.infoHub.kpiservice", "tenantId":"3fed53d0-9817-11e9-bc42-526af7764f64", "timestampEventOccurred":"2019-06-26T13:40:16+0000:00", "timestampEventReceived":"2019-08-21 09:33:00", "eventDetails":{ "rule":{ "id":"kpi-docVolume-kpi-rule", "name":"kpi-docVolume-kpi-rule" }, "dimensionValues":[ { "name":"entityType", "value":"document" }, { "name":"dateKey", "value":"2019-06-26" } ], "values":[ { "name":"documentVolume", "value":"27" } ] } }
- Window event
- The preceding delta event can be aggregated to generate a longer period event, which is called a
window event. The window events are recorded in ES according to different dimension keys. If
sendDataWhenNoData=yesis set in the KPI rule, then these window events can be queried according to the window type of this KPI rule and then sent out.- For hour window type, it runs immediately as the service started. And it queries the per-hour window results then sends to Kafka.
- For day window type, it will run after an initial delay (to 01:00:00). And then it will run in a one-day cycle.
- For month and year window type, it runs the same way as the day window type. However, it will only do the real task when it is the first day of that per-month. Or the first day of that per year.
Note: Currently, if it has onetenantIddimension and thesendEventsWhenNoData=yes, then it sends the empty events for these tenants.- If the current KPI rule does not specify the tenant, then it sends empty events for all tenants.
- If the current KPI rule specifies the tenants, then it sends only empty events for these tenants. The following example shows the demo empty window event:
{ "eventCode":"businessEvent", "objectType":"com.ibm.infohub.kpi.service.windowEvent", "sourceId":"com.ibm.infoHub.kpiService", "tenantId":"4a87c4b2-b38a-4270-85cf-1ea6fb7e9d63", "timestampEventOccurred":"2019-10-18T13:00:00", "timestampEventReceived":"2019-10-18T14:24:50", "eventDetails":{ "rule":{ "id":"a67b015e-106c-4336-b30d-eba888888", "name":"kpi-count-test-kpi-rule" }, "dimensionValues":[ { "name":"hour", "value":"2019-10-18T13" } ], "values":[ { "name":"quantityValue", "value":0 } ] } }