Introduction

Kafka is a distributed and scalable publish-subscribe messaging system and a robust queue that can handle a high volume of data and enables you to pass messages from one end-point to another.

Sterling B2B Integrator has introduced visibility improvements using Kafka. At present, the Visibility solution has the following limitations:

Lack of end-to-end visibility for file transfer
Performance issue with DMI visibility

Why Kafka?

By integrating with Kafka, you can overcome the above limitations. Sterling B2B Integrator publishes events to Kafka and you can read or consume events from Kafka. Integration with Kafka provides end-to-end visibility for file transfer and flexibility to create your own Visibility dashboard.

Sterling B2B Integrator publishes events to Kafka in a phased manner. EDI correlations are published to Kafka in the first phase.

Publishing EDI Correlation to Kafka

EDI publishes numerous key-value correlations during enveloping and de-enveloping. These correlations are used for EDI correlation search, tracking status of the document, checking overdue acknowledgments, and so on.

These correlations cause a serious database performance degradation issue, which requires purging of correlation data on a regular basis. The purging of data leads to deletion of data, which makes tracking EDI transactions or document statuses impossible.

To overcome this, Sterling B2B Integrator publishes correlation data to Kafka. In the current release, correlation data for standard X12 and EDIFACT are published to Kafka.

You can also track document statuses, since correlation data can be stored for a longer duration.

This offers the following benefits:

Reduces load on the database, which regulates performance degradation
Provides flexibility to create Visibility dashboards

Since the EDI correlation data is moved out of the database, the following features are not available as part of the Sterling B2B Integrator dashboard:

EDI Correlation Search
Overdue acknowledgment status
EDI reports
Manual update of Interchange/Group ACK status

You must consume the events from Kafka to build the above functionalities and any other Visibility dashboard.

Event Schema for Kafka EDI correlation events

Field	Type	Mandatory or Optional	Description
eventid	String	Mandatory	Unique Id for event (UUID)
Category	String	Mandatory	ADEVENT, BPEVENT, CORRL, DOCEXT, AUEVENT, FGEVENT
type	String	Optional	Type of event. Existing event name in Sterling B2B Integrator. For example: CommSessionUpdate, CommConnect, CommAuthentication, CommXfileBegin
stage	String	Optional	Event stage. This is new field introduced to describe the event stage. For example: Session establishment or connection event, File transfer, Routing, BP execution. Not used as of now.
scope	String	Mandatory	Component name (EDI for EDI correlation). This is an existing scope value wherever available. For example: Aft, System, EDI, Document
datacenterid	String	Optional	Data center Id through which the event is generated. Used in a multi data centre environment. Global Mailbox.
datacentername	String	Optional	Data center name. Used in multi data centre environment. Global Mailbox.
clustername	String	Optional	Cluster name in data center. As of now there is no cluster name but will be introduced in future.
nodename	String	Mandatory	Sterling B2B Integrator node name. For example: node 1, node 2.
level	Enum	Mandatory	Event Level (Normal, Warning, Debug, Error, Fatal)
correlationlevel	Enum	Optional	Correlation Level (Interchange, Group, Transaction, Document, System, and so on).
creationtime	Long	Mandatory	Timestamp when the event is generated.
trackingid	String	Optional	Tracking Id is used for correlation. Empty in case of connection/session events. Transfer/BP/correlation must have tracking id. For Correlation events in EDI, `enveloping.properties.TRACKING_LEVEL=full` is required.
documentid	String	Optional	Document Id
parentdocumentid	String	Optional	Parent document Id. For example: In case of EDI processing, Interchange and Group Level are 2 documents. The Group Level document has Interchange document as parentdocumentid.
documentname	String	Optional	Name of the file that is transferred.
documentsize	Integer	Optional	Size of the file transferred/document size
mailboxtype	String	Optional	Location where the document is stored.
protocolstandard	String	Optional	Protocol used in transfer/Standards name for correlation of EDI/Service name/Empty.
direction	Enum	Optional	Direction of the file transfer (INBOUND/OUTBOUND)
workflowid	Integer	Optional	BP workflow Id.
parentworkflowid	Integer	Optional	Workflow Id of the parent BP, if any. This helps to correlate a schedule transfer initiated by executing a BP.
status	String	Mandatory	File transfer status.
senderpartnerid	String	Optional	Sender partner ID
senderpartnername	String	Optional	Sender partner name (Organization name)
senderuserid	String	Optional	Sender user ID
receiverpartnerid	String	Optional	Receiver partner ID
receiverpartnername	String	Optional	Receiver partner name
receiveruserid	String	Optional	Receiver user id
islocallyinitiated	Boolean	Optional	True/false
sessionid	String	Optional	Transfer session Id
schemakey	String	Mandatory	Schema key as per Sterling B2B Integrator, which can be used to filter the events. Existing schema key values, for example, Aft.Visibility.CommSessionUpdate.
additionalparams	List	Optional	List containing any additional parameters for the event. Each of the parameter in the set is of the form (paramid, paramkey, paramvalue).

The correlation key-value pairs are published in the additionalparams field of the events. Multiple events are generated in a single document transfer in EDI, based on the number of correlations published at various steps.

Sample EDI correlation in JSON format


Events generated in EDI X12 enveloping flow:
{ 
  "category":"CORRL",
  "type":null,
  "stage":null,
  "scope":"EDI",
  "level":"Normal",
  "creationtime":1588853317123,
  "direction":"OUTBOUND",
  "status":"",
  "eventid":"664028171ef058612node1",
  "correlationlevel":"EDI",
  "datacenterid":null,
  "datacentername":null,
  "clustername":null,
  "nodename":"node1",
  "trackingid":"594014171ef058612node1",
  "documentid":"674001171ef058612node1",
  "parentdocumentid":null,
  "documentname":"X12OutboundReconcileTest_input.txt",
  "documentsize":953,
  "mailboxtype":null,
  "protocolstandard":"EDI-X12",
  "parentworkflowid":null,
  "workflowid":68038,
  "senderpartnerid":"ReconTest5Snd",
  "senderpartnername":null,
  "senderuserid":null,
  "receiverpartnerid":"ReconTest5Rcv",
  "receiverpartnername":null,
  "receiveruserid":null,
  "islocallyinitiated":false,
  "sessionid":null,
  "schemakey":"CORRL.EDI",
  "additionalparams":[ 
    { 
      "id":"72",
      "name":"RequiresEnvelope",
      "value":"1"
    },
    { 
      "id":"73",
      "name":"EnvelopeID",
      "value":"X12OutboundSndToRcvRecTest05_ST"
    },
    { 
      "id":"221",
      "name":"EDIEncoderVersion",
      "value":"6100"
    },
    { 
      "id":"169",
      "name":"DocumentTime",
      "value":"0001588853315715"
    }]
}
{
  "category":"CORRL",
  "type":null,
  "stage":null,
  "scope":"EDI",
  "level":"Normal",
  "creationtime":1588853319719,
  "direction":"OUTBOUND",
  "status":"",
  "eventid":"144254171ef058612node1",
  "correlationlevel":"Transaction",
  "datacenterid":null,
  "datacentername":null,
  "clustername":null,
  "nodename":"node1",
  "trackingid":"594014171ef058612node1",
  "documentid":"854076171ef058612node1",
  "parentdocumentid":null,
  "documentname":"ISA",
  "documentsize":893,
  "mailboxtype":null,
  "protocolstandard":"EDI-X12",
  "parentworkflowid":null,
  "workflowid":68039,
  "senderpartnerid":"ReconTest5Snd",
  "senderpartnername":null,
  "senderuserid":null,
  "receiverpartnerid":"ReconTest5Rcv",
  "receiverpartnername":null,
  "receiveruserid":null,
  "islocallyinitiated":false,
  "sessionid":null,
  "schemakey":"CORRL.EDI",
  "additionalparams":[
    {
      "id":"20",
      "name":"TransactionSenderID",
      "value":"ReconTest5Snd"
    },
    {
      "id":"21",
      "name":"TransactionReceiverID",
      "value":"ReconTest5Rcv"
    },
    {
      "id":"22",
      "name":"TransactionControlNumber",
      "value":"1012"
    },
    {
      "id":"24",
      "name":"TransactionVersion",
      "value":"003050"
    },
    {
      "id":"26",
      "name":"TransactionComplianceStatus",
      "value":"OK"
    }]
}