Contribute in GitHub:

Manta Event System

Introduction

Manta Event System is a notification and a messaging system that produces and processes events from all applications in the IBM Manta Data Lineage ecosystem. Manta events are messages with a clearly defined format that describe and inform users of occurrences of significance in the system. An event can, for example, inform users that a workflow has been successfully or unsuccessfully finished or provide success statistics for an analytical scenario. Users can consume events using one of the external connectors (for example, a connector to Kafka) and implement their own integrations over the events.

Manta Event

A Manta event is an abstract representation of something that happened in the IBM Manta Data Lineage ecosystem. It consists of a group of attributes describing the event and the conditions under which it was generated, including a message body which describes the event in a human-understandable way. Events can be considered to be important logs (of any severity) that are useful to show and process separately. Furthermore, an event contains a set of parameters, unique to each event, which a user can use to implement their own integrations. For example, the success rate of an analytical scenario may be a parameter of an event and a user may implement logic where something would happen in their own internal system if this success rate dropped below a designated threshold. The following sections will define some event attributes that will always have one of the explicitly enumerated values.

Event Type

Event type is an attribute of an event indicating its meaning and purpose.

INFORMATION

Events of this type describe any events that are expected to occur in the correctly functioning environment. An example of such an event might be that a scenario has finished successfully.

ISSUE

Events of this type describe any events that point to unexpected or erroneous behavior in the environment. An example of such an event might be that a scenario has failed.

Event Priority

Event priority is an attribute of an event indicating its severity or importance. This attribute can have any of the following values.

LOW

MEDIUM

HIGH

Event message Format

Events are transported as messages within a distributed system. The payload of these messages is in JSON format and consists of the parameters that are defined in the following table.

Parameter name	Type	Description
`name`	String	Name of the event
`timestamp`	Long	UNIX timestamp with milliseconds of the event generation
`type`	String	An enumeration value describing the event type
`priority`	String	An enumeration value describing the importance of the event
`message`	String	A human-readable message describing the event
`attributes`	Object	A group of key-value pairs, which are defined uniquely per event
`source`	Object	A group of attributes describing the source of the event
`source.name`	String	Name of the source (Workflow Execution, Orchestration API, and so on)
`source_id`	String	ID of the source execution workflow for executions started from Manta Process Manager
`application`	String	Name of the application generating the event (Manta Flow Server, Manta CLI, and so on)
`workflowExecution`	Object	Optional attribute group describing the workflow/scenario execution
`workflowId`	String	Name of the workflow as defined in Manta Process Manger
`scenario`	String	Name of the run scenario
`technology`	String	Name of the technology of the scenario
`phase`	String	Name of the phase of the scenario (Extraction, Analysis, and so on)

All the values for an event instance are transported as a JSON object, which is defined as follows.

{
  "name": "string",
  "timestamp": "long",
  "type": "string",
  "priority": "string",
  "message": "string",
  "attributes": {
    "key": "value"
  },
  "source": {
    "name": "string",
    "sourceId": "string",
    "application": "string",
    "workflowExecution": {
      "workflowId": "string",
      "scenario": "string",
      "technology": "string",
      "phase": "string"
    }
  }
}

Example

The definitions for the event format can be used like in the following example.

{
  "name": "SCENARIO_SUCCESS_EVENT",
  "timestamp": 1632410653861,
  "type": "INFORMATION",
  "priority": "HIGH",
  "message": "Scenario 'Diagnose Repository Scenario' successfully finished its execution.",
  "attributes": {
    "sceanrio": "Diagnose Repository Scenario"
  },
  "source": {
    "name": "Workflow Execution",
    "source_id": "15",
    "application": "MANTA CLI",
    "workflowExecution": {
      "workflowId": "Test Workflow",
      "scenario": "Diagnose Repository Scenario",
      "technology": "MANTA",
      "phase": "Extraction"
    }
  }
}

External Connectors

An external connector from the Manta Event System is a user-configurable connection to a third-party system to which events generated while running various tasks in Manta Data Lineage are routed. Users might implement their own integrations and logic for various systems that they might use internally.

Here is a list of the external connectors that are currently supported.