Mapping similar ticket data from ServiceNow

If you need to use historical incidents from ServiceNow as similar ticket data, you need to map the data to an expected format and insert the data into Elasticsearch.

Historical incidents are required to first be inserted into Elasticsearch for similar ticket data. The data format consists of one incident record per line, in JSON format, with the original raw data. Raw data must be normalized to the IBM Cloud Pak for Watson AIOps target format. This normalization includes mapping fields, converting date-time formatting, and more. Incidents are then inserted in batch. When the insertion is done, IBM Cloud Pak for Watson AIOps tries a simple search to make sure everything runs as expected.

For more information about training similar tickets models in IBM Cloud Pak for Watson AIOps, see Setting up training for similar tickets.

Raw similar ticket data

IBM Cloud Pak for Watson AIOps ingests incidents from standard or custom data sources, for the incidents to be analyzed for similarities and action recommendations. Offline historical raw incidents are first uploaded to the Elastic database to extract indexes and set the stage for finding incidents that are similar to incoming live incident data.

To maintain the structure of the incident data in the Elastic database, you must transform raw incident data into a normalized format. That normalized data can then be used in the training pipeline. For example, the following raw JSON incident data comes from ServiceNow:

{"number": "INC0010066", "short_description": "Containers being killed in ts-payment-mongo service. Users may notice issues during payment", "description": "", "sys_created_on": "2020-01-08T22:48:23Z", "sys_updated_on": "2020-01-08T22:48:23Z", "resolved_at": "2019-12-11T07:36:00Z", "closed_at": "2019-12-11T07:36:00Z", "business_duration": "", "severity": 3, "priority": 5, "impact": 3, "state": "closed", "comments_and_work_notes": "", "close_notes": "This issue has been resolved by the DBA team, which SQL queries were updated to reboot proxy agent to prevent the monitoring notification system from failing.", "parent_incident": ""}

In this example, you can extract data from ServiceNow, then map that raw incident data to the IBM Cloud Pak for Watson AIOps normalized output schema. The source and format of your exported data doesn't matter. That data must map to the IBM Cloud Pak for Watson AIOps normalized output data schema for training.

For more information about exporting data from ServiceNow, see Exporting data Opens in a new tab.

For more information about managing incident data in ServiceNow, see Incident management Opens in a new tab.

Normalized similar ticket data

The following sample illustrates the result of mapping data from an external source for use with IBM Cloud Pak for Watson AIOps:

{
  "application_group_id": "1",
  "application_id": "1",
  "timestamp": 1570221600,
  "utc_timestamp": "2019-10-04T20:40:00.038Z",
  "type": "incident",
  "incident": {
    "incident_id": "fed6b7bf-f741-49d1-ab36-5d04e9a5c128",
    "title": "Users are not able to access Discovery and Conversation in EU",
    "description": "Unable to connect to email using my laptop.",
    "created_at": "2019-09-11T06:39:48.000Z",
    "updated_at": "2019-09-11T07:57:58.000Z",
    "resolved_at": "2019-09-11T07:57:58.000Z",
    "closed_at": "2019-09-11T07:57:58.000Z",
    "started_at": "2019-09-11T07:57:58.000Z",
    "business_duration_ms": 15347,
    "severity": 1,
    "priority": 2,
    "impact": 3,
    "state": "new",
    "source": {
      "source_name": "ServiceNow",
      "source_url":"https://ibm.servicenow.com/INC0000060",
      "source_incident_id": "INC0000060",
      "source_application_id": "tokenizer124"
    },
    "comments": [
      {
        "comment_text":"Rebooted it."
      }
    ],
    "related_incidents":[
      {
        "source_incident_id":"INC0000059",
        "relationship":"parent"
      }
    ],
    "resolution": {
      "rca_id": "RCA00013547",
      "resolution_summary": "The problem was resolved by rebooting the laptop."
    },
    "features": [
    ]
  },
  "meta_features": []
}

Description of similar ticket normalized attributes

Required attributes expected as input to the pipeline
Attribute Description
title Short description or title of the incident.
description Long description of the incident.
timestamp/utc_timestamp Epoch timestamp of the event in the log entry and the utc_timestamp formatted as yyyy-mm-ddTHH:MM:SSZ.
created_at Coordinated Universal Time at which the incident was created formatted as yyyy-mm-ddTHH:MM:SSZ.
updated_at Coordinated Universal Time at which the incident was last updated formatted as yyyy-mm-ddTHH:MM:SSZ.
resolved_at Coordinated Universal Time at which the incident was resolved formatted as yyyy-mm-ddTHH:MM:SSZ.
closed_at Coordinated Universal Time at which the incident was closed formatted as yyyy-mm-ddTHH:MM:SSZ.
started_at Coordinated Universal Time at which the incident started formatted as yyyy-mm-ddTHH:MM:SSZ.
severity Severity of the issue (integer)
priority Priority of the issue (integer)
impact Impact of the issue (integer).
source_name Data source in which the incident is stored (for example, ServiceNow).
source_url URL of the incident in the source.
source_incident_id incident_id from the source (for example, NUMBER in ServiceNow).
source_application_id Application from which the incident came (for example, the configuration item in ServiceNow).
comments Comments or closed notes that talk about the list of actions that are taken, or comments that are added by the user who is attending to the incident.
related_incidents Any related incidents, such as parent incidents, child incidents, or CIE documentation that is attached with the incident.
resolution Any RCA ID or resolution that is provided for the incident.
Optional attributes that might add value
Attribute Description
business_duration Duration for which the incident was alive without resolution.
Static attributes
Attribute Description
application_group_id The application_group_id to support multitenancy. This value is provided as part of the initial configuration.
application_id Application that IBM Cloud Pak for Watson AIOps is monitoring. This value is provided as part of the initial configuration.
type This value is hardcoded to the incident.
timestamp Epoch time of the incident normalization, in seconds.
utc_timestamp Normalization time in the Coordinated Universal Time time zone.
incident_id The unique UUID.
state Current state of the incident.
Attributes analyzed and populated in the pipeline
Attribute Description
features Placeholder for any features.
meta_features Placeholder for service analysis.