Key concepts

Edit online

The following diagram demonstrates how the key concepts work together.

Event Endpoint Management key concepts

Catalog

The catalog lists all published and archived virtual topics. Users in your organization can browse the virtual topics, and view their descriptions, schemas, tags, and other details. From the catalog, the user can subscribe to a published virtual topic through an Application.

Authors can use the catalog to check what virtual topics are published.

Cluster

Kafka runs as a cluster of one or more servers (Kafka brokers). The load is balanced across the cluster by distributing it among the servers. The term cluster in this documentation refers exclusively to Kafka clusters.

Cluster maintainers

Cluster maintainers can edit and maintain cluster connection definitions for the clusters that they are assigned to. Cluster maintainers cannot add topics to Event Endpoint Management from the cluster, change the maintainers of the cluster, or delete the cluster.

Controls

Controls that manage how the event data is accessed, presented, and processed can be added to your virtual topics.

The following types of controls are available: event data controls and security controls.
Event data controls
With event data controls, you can manage and modify the event data that is sent to or received from the virtual topic.

You can add the following event data controls to a consume-enabled virtual topic:

  • Content filtering: Control which events are delivered to subscribers based on event data or subscriber data.
  • Redaction: Redact sensitive data from events.
  • Schema filtering: Filter out events that do not comply with the AVRO or JSON schema that is defined for the virtual topic.
  • Quota enforcement: Manage the rate at which data is consumed from your virtual topic.

You can add the following event data controls to a produce-enabled virtual topic:

  • Schema enforcement: Allow only events that comply with the AVRO or JSON schema that is defined for the virtual topic.
  • Quota enforcement: Manage the rate at which data can be written to your virtual topic.
Security controls

With security controls, you can manage how the virtual topics are secured.

You can add the following security controls to both consume-enabled and produce-enabled virtual topics:

  • Approval: Use the approval control to force users to request access to your virtual topic.
  • Mutual TLS: Require users to present a valid client certificate that matches on common name and other specified subject fields.
  • OAuth 2.0: Require users to provide their OAuth2 credentials to access the virtual topic.
  • SASL credentials: Require users to provide their SASL credentials to access the virtual topic.

Consume

A client can consume events from virtual topics that are created from consume-enabled source topics. The client subscribes to the virtual topic through an application, and can then consume Kafka events from the virtual topic by using standard Kafka client libraries.

Event

An event represents a meaningful occurrence or change in the state of a system or a value.

Event endpoint

Event endpoints refer to the published virtual topics that are available to Kafka clients through the Event Gateway.

IBM API Connect Developer Portal

IBM API Connect Developer Portal is a developer portal integrated in IBM webMethods Hybrid Integration where you can publish virtual topics. When you publish virtual topics to Developer Portal, users can discover and subscribe to virtual topics through a unified interface that includes both synchronous and asynchronous APIs.

Virtual topic

A virtual topic represents a Kafka topic that the Event Endpoint Management administrator adds to Event Endpoint Management, and is available to be published to the Event Endpoint Management catalog. When you publish virtual topics to the Event Endpoint Management catalog, you make them available to the clients through an Event Gateway. A virtual topic is also referred to as an Event endpoint from the Kafka client's perspective.

Event Gateway

Access to virtual topics is managed by one or more Event Gateways. The Event Gateway handles the incoming requests from clients to access a virtual topic, routing traffic securely between the Kafka cluster and the client.

The Event Gateway is independent of your Kafka clusters, making access to topics possible without requiring any changes to your Kafka cluster configuration.

Event Gateways are not deployed inside IBM webMethods Hybrid Integration, they are deployed in your own environment, ideally close to your Kafka clusters for optimal performance.

Event Gateway group

An Event Gateway group is a logical group of Event Gateway instances, to which virtual topics are published.

Example gateway groupings:

  • Event Gateways that are deployed to manage Kafka traffic to designated Kafka clusters.
  • Event Gateways that are co-located within the same geographical area.
  • Event Gateways that manage traffic for different Kafka topics that are at different stages of maturity. For example, development, test, or production.

When deploying an Event Gateway, you must specify the gateway group that it belongs to.

When a new Event Gateway starts, it contacts the Event Manager, which responds with all the virtual topics that are published to the gateway’s group. Kafka clients can then access the published virtual topics through the new Event Gateway.

Key points:

  • You must have at least one gateway group.
  • An Event Gateway can be a member of only one gateway group.

Event Manager

The Event Manager component adopts your Kafka topics into a central location after you describe them as source topics, and you can then generate virtual topics for sharing with others. The Event Manager provides features to configure how you want to present the topic data by using virtual topics, which can be accessed through the Event Gateway.

The Event Manager is deployed in your IBM webMethods Hybrid Integration environment.

Source topic

A source topic is a topic that is on a Kafka cluster, which you make available in Event Endpoint Management. For a single source topic, you can define one or more virtual topics that provide different security and event data controls for that source topic. Where there is no risk of ambiguity, the term topic is used to refer to a source topic in this documentation.

Topic

A topic refers to a Kafka topic on a Kafka cluster. The Event Endpoint Management administrator adds topics from Kafka clusters, which can then be exposed to clients on an Event Gateway. Where there is the risk of ambiguity, topics are referred to as source topics in this documentation.

Topic editors

Topic editors can update source topic information and manage virtual topics for the source topics that they are assigned to. Topic editors cannot change the editors of the source topic or delete source topics.

Message

The unit of data in Kafka. Each message is represented as a record, which comprises two parts: key and value. The key is commonly used for data about the message and the value is the body of the message. Message is also sometimes referred to as event data and record.

To learn more about key concepts, see the Apache Kafka documentation.

Virtual topic viewers

Virtual topic viewers are a group of users that can view and subscribe to selected virtual topics in the catalog, in addition to existing public virtual topics. Users can view and subscribe to the virtual topics that are assigned to their user group.

Owner

A user with the author role who can assign user groups to view virtual topics, edit source topics, and maintain clusters that they create.

Produce

A Kafka client can write events to a virtual topic that is published to the catalog. The client must subscribe to a virtual topic on a produce-enabled source topic, and then they can produce Kafka events to the virtual topic through the Event Gateway, by using standard Kafka client libraries.

Subscription

A subscription is a set of credentials that can be used by a Kafka client to access the virtual topics that are included in an application.

User groups

A group of users that have access to specific actions such as viewing virtual topics, editing source topics, and maintaining clusters. User groups are defined in the IBM SaaS Console by an admin for IBM webMethods Hybrid Integration. For more information, see managing user groups.

Application

An application in Event Endpoint Management is set of credentials that can be used by one or more Kafka clients to access one or more virtual topics. When you want to provide access to a virtual topic, you create an application in the Event Endpoint Management UI, and then subscribe the application to the virtual topic.