PagerDuty entity extraction mapping files

When you create a PagerDuty connection, you can provide a mapping file to IBM Cloud Pak for Watson AIOps that enables entity extraction for alerts.

Entity extraction mapping files detail the format of different alerts, from various data systems, and are formatted in JSON. Given an input JSON file, and a mapping JSON file, IBM Cloud Pak for Watson AIOps extracts relevant entities of interest for output. The data is processed in each action, and then passed on to the next action.

Entity extraction mapping files are uploaded to IBM Cloud Pak for Watson AIOps when you create a PagerDuty event connection. For more information about creating the connection to integrate with PagerDuty, see Creating PagerDuty connections.

Important: To create entity extraction mapping files, you must understand the various alert formats that your organization uses. You must be able to identify the important entities mentioned in the following alert examples.

The mapping format is a simple (type, field, condition) structure, in all levels. The following example shows the base structure:

[
    {
        "type": "query action",
        "field": "json jq style field name",
        "condition": "parameter to action",
        "rules": [
            {
                "type": "rule action",
                "field": "json jq style field name",
                "condition": "parameter to action",
                "filters": [
                    {
                        "type": "filter action",
                        "field": "json jq style field name",
                        "condition": "parameter to action",
                        "actions": [
                            {
                                "seqId": sequence number,
                                "type": "data action",
                                "field": "json jq style field name",
                                "condition": "parameter to action"
                            }
                        ]
                    }
                ]
            }
        ]
    }
]

Actions

The preceding example mapping format includes the four action steps:

query action
rule action
filter action
data action

Query action and rule action act like if statements. Those actions check whether the query and rule must be run. These actions decide whether the input source needs to be processed. Filter action and data action act like data extractors that take data from the input source. The filter action applies the same data to every rule, and the data action (in sequence) processes the data to the final output.

For examples of mapping instances, see Example mapping instances.

Action syntax

Table 1. Action syntax
Syntax	Description
`true`	Pass the condition for `query action` and `rule action`; no condition is required.
`truncate_from`, `truncate_to`, and `truncate_from_to`	Truncate array `from`, `to`, or `from_to`. The condition integer for `from` and `to`, and array for `from_to`.
`join`	Join array with a character in condition.
`jsonify`	Convert the string to `JSON`; no condition is required.
`jq`	`JSON` query. The condition is in a `jq` (Linux command) format, for instance, `".body.cef_details.description"`.
`split`	Split string into array. The condition can be a regular expression and can break up multiple characters.
`strip`	Strip the string with a regular expression in the condition.
`pairs`	Makes an array into `JSON`, for example, `[a1, a2]` becomes `{ a1: a2}`.
`replace`	Replace string with other words.
`keymap`	Map string or array to `json` format. When an output is `B` and a condition is `A`, then the final output is `{ A: B }`. Also, when output is `[C, D]` and condition is `[A, B]`, then the final output is `{ A:C, B, D }`.
`regex`	Regular expression on the data. The condition includes the regular expression.
`urls`	Return all URLs in the data.
`ips`	Return all IP addresses in the data.
`key`	Find on item in a `JSON` format.
`select_one` or `select_all`	Select first key item or all key items in the `JSON` format.

Output schema

The following table lists the entity types of interest, and the expected key values of the JSON output by using the standard names:

Entity Type	Key Value
Node ID	`NodeReference`
Pod ID	`PodReference`
Application ID	`ApplicationID`
Kubernetes Deployment Name	`kubernetes_deployment_name`
Backend Service Name	`backend_service_name`
Kubernetes Namespace Name	`kubernetes_namespace_name`
URLs	`urls`
IP Address	`ip`

Note: Use replace to change the existing key from backend.service.name to backend_service_name.

Example output

[
    {
        "type": "true",
        "rules": [
            {
                "type": "jq",
                "condition": ".body.cef_details.details",
                "filters": [
                    {
                        "type": "true",
                        "actions": [
                            {
                                "seqId": 1,
                                "type": "key",
                                "condition": "backend.service.name"
                            },
                            {
                                "seqId": 2,
                                "type": "replace",
                                "condition": ["backend.service.name", "backend_service_name"]
                            }
                        ]
                    }
                ]
            }
        ]
    }
]

Example mapping instances

The following examples demonstrate the JSON mapping file formatting for specific instances.

Example: Extracting key value pairs from an input file

Given the following JSON input file:

{
    "type" : "log-anomaly-detection",
    "application_group_id" : "123abc",
    "application_id" : "123abc",
    "alert" : {
        "log_anomaly_data" : {
            "detected_at" : 1582197546993,
            "source_application_id" : "ts-assurance-mongo"
        },
        "entity" : {
            "pod" : [
                {
                    "counts" : 4,
                    "name" : "test-pod-1-id"
                }
            ]
        }
    },
    "timestamp" : 1582197546993
}

The mapping file to extract the source_application_id and pod names can resemble the following example:

[
    {
        "type": "true",
        "rules": [
            {
                "type": "true",
                "filters": [
                    {
                        "type": "true",
                        "actions": [
                            {
                                "seqId": 1,
                                "type": "jq",
                                "condition": ".alert.log_anomaly_data.source_application_id"
                            },
                            {
                                "seqId": 2,
                                "type": "keymap",
                                "condition": "ApplicationID"
                            }
                        ]
                    },
                    {
                        "type": "true",
                        "actions": [
                            {
                                "seqId": 1,
                                "type": "jq",
                                "condition": ".alert.entity.pod[].name"
                            },
                            {
                                "seqId": 2,
                                "type": "keymap",
                                "condition": "PodReference"
                            }
                        ]
                    }
                ]
            }
        ]
    }
]

In this example, type is set to true to not restrict any condition. The filters action acts as the data extractor. Under actions, the file first extracts the field value given the field key .log_anomaly_data.source_application_id as the condition. It then passes the value to the next sequence to specify the output format.

Example output:

{
    "ApplicationID": "ts-assurance-mongo",
    "PodReference": "test-pod-1-id"
}

Example: Extra rule capabilities

The syntax of the mapping also allows regular expression capabilities.

Given an example alert in the following JSON format:

{
    "alert_key": "3231",
    "body": {
        "cef_details": {
            "client": "abc",
            "client_url": "https://jp-tok.monitoring.cloud.ibm.com:443/#/events/notifications/l:604800/3231/details",
            "contexts": [
                {
                    "href": "https://jp-tok.monitoring.cloud.ibm.com:443/#/alerts/8681",
                    "text": "Link to Alert definition that triggered the event",
                    "type": "link"
                },
                {
                    "href": "https://jp-tok.monitoring.cloud.ibm.com:443/#/events/notifications/l:604800/3231/details",
                    "text": "Troubleshoot Event",
                    "type": "link"
                }
            ],
            "dedup_key": "3231",
            "description": "TEST ALERT: Testing Notification Channel Assistant clu",
            "details": {
                "Alert ID": 8681,
                "Alert description": "Alert description",
                "Alert name": "TEST ALERT: Testing Notification Channel Assistant clu",
                "Condition": "avg(cpu.used.percent) > 85",
                "Event ID": 3231,
                "Scope": "host.mac = \"08:00:27:70:1a:03\"",
                "Segment": "container.name = 'container1_0' and host.mac = '08:00:27:70:1a:03'",
                "Severity": "Medium",
                "Source": "Sysdig",
                "Timestamp": 1561485893650,
                "UTC": "2019-06-25T18:04Z",
                "Username": "06b5cb929d554e6507b5828af6ecc2e5_d38fb44b-0c0a-404c-858a-7d0e848bb832_ibmid-310000dpng@ibm.com",
                "Value": "90.8136752422863"
            },
            "message": "TEST ALERT: Testing Notification Channel Assistant clu",
            "mutations": [],
            "version": "1.0"
        }
    },
    "integration": {
        "summary": "Sysdig",
    } 
}

The mapping file to extract the container and host.mac entities from the Segment field can resemble the following example.

{
    "type": "regex",
    "field": ".integration.summary",
    "condition": "sysdig",
    "rules": [
        {
            "type": "jq",
            "condition": ".body.cef_details.details.Scope",
            "filters": [
                {
                    "type": "true",
                    "actions": [
                        {
                            "seqId": 1,
                            "type": "split",
                            "condition": " and "
                        },
                        {
                            "seqId": 2,
                            "type": "split",
                            "condition": " in | = "
                        },
                        {
                            "seqId": 3,
                            "type": "strip",
                            "condition": " \"\\'()\t\r\n"
                        },
                        {
                            "seqId": 4,
                            "type": "pairs"
                        },
                        {
                            "seqId": 5,
                            "type": "replace",
                            "condition": [".", "_"]
                        }
                    ]
                }
            ]
        },
        {
            "type": "jq",
            "condition": ".body.cef_details.details.Segment",
            "filters": [
                {
                    "type": "true",
                    "actions": [
                        {
                            "seqId": 1,
                            "type": "split",
                            "condition": " and "
                        },
                        {
                            "seqId": 2,
                            "type": "split",
                            "condition": " in | = "
                        },
                        {
                            "seqId": 3,
                            "type": "strip",
                            "condition": " \"\\'()\t\r\n"
                        },
                        {
                            "seqId": 4,
                            "type": "pairs"
                        },
                        {
                            "seqId": 5,
                            "type": "replace",
                            "condition": [".", "_"]
                        }
                    ]
                }
            ]
        }
    ]
}

This mapping first sets the .integration.summary to be Sysdig, as this mapping set is only applicable when the alert summary is from Sysdig. The sequence of actions specifies the processing of the container.name = 'container1_0' and host.mac ='08:00:27:70:1a:03' to extract the container and host.mac entities.