Managing events by using Apache Kafka

You can use Apache Kafka to manage events that are created by IBM® InfoSphere® Information Server. Use this open source infrastructure to react to events such as workflow engines or emails and to create your own events.

About this task

Apache Kafka is an open source, publish-and-subscribe messaging system that is built for high throughput, speed, availability, and scalability.

Kafka topics are feeds of messages in categories. Processes that publish messages to a topic are called producers. Processes that subscribe to topics and process the feed of published messages are called consumers. For details and examples, see consumer sample and producer sample.

InfoSphere Information Server has a ready-to-use installation of Kafka and a Kafka topic that provides all InfoSphere Information Server events as Kafka messages. You can see all messages from all suite products, and you can consume these messages with any Kafka client implementation.

A command-line consumer directs messages to a command window. You use the consumer to see messages that are created by InfoSphere Information Server. Kafka uses the Zookeeper API, which is a centralized service that maintains configuration information.

Kafka message consumers are not limited to capturing messages from InfoSphere Information Server. You can also retrieve messages from your own applications when you implement your own Kafka producer.

Procedure

Start a command-line consumer to inspect messages from InfoSphere Information Server.
1. Determine the Zookeeper connection string. In a command window, go to the directory and enter the commands:
  UNIX operating system
```
cd <IS_Install_path>/ASBServer/bin
./iisadmin.sh -display -key com.ibm.iis.sdp.zookeeper.connect
```
  Microsoft Windows operating system
```
cd <IS_Install_path>\ASBServer\bin
iisadmin -display -key com.ibm.iis.sdp.zookeeper.connect
```
  The command returns the Zookeeper connection string in the form: hostname:port.
2. Start the command line consumer. In a command-line window, go to the directory and enter the commands:
  UNIX operating system
```
cd <IS_install_path>/shared-open-source/kafka/install/bin
./kafka-console-consumer.sh --zookeeper “hostname:port” --topic “InfosphereEvents” –from-beginning
```
  Microsoft Windows operating system
```
cd <IS_install_path>\shared-open-source\kafka\install\bin\windows
kafka-console-consumer.bat --zookeeper “hostname:port” --topic “InfosphereEvents” –from-beginning
```
  Note hostname:port is the connection string that you got in the previous step. InfosphereEvents is the name of the Kafka topic that is used by InfoSphere Information Server. The console prints all InfoSphere Information Server messages.
  
  When you run this command, all events that are generated in InfoSphere Information Server are displayed in the command-line window.
For example, if you create an information governance rule in InfoSphere Information Governance Catalog, the following message is created:
```
{
“eventType”:”IGC_NEWRULE_EVENT”
“applicationType”:”Information Governance Catalog”
"ASSET_TYPE":"Information Governance Rule",
"ASSET_RID":"6662c0f2.e1b13efc.8gi55ct6c.l1eh0ds.3ttrq2.dueampgjs67s528k6b1el",
"ASSET_CONTEXT":"",
"ACTION":"CREATE",
"ASSET_NAME":"newrule"
}
```
Tip: Each message is a JSON object. When you use a Kafka client API to read messages from the InfosphereEvents topic, you can use JSON APIs to process the messages. If you choose to send your own messages to the InfosphereEvents Kafka topic, use the same message format that is in this example to simplify your consumer implementation.
Implement a Kafka consumer for InfoSphere Information Server messages.
1. Design your own consumer. You can use the sample application to render statistical charts for all InfoSphere Information Server events. You can also use the sample application to launch business process definitions from IBM Business Process Manager. You can also write a custom application with Node.js to react on Kafka events by using the Information Analyzer Notifications sample.
2. Run your consumer. Use the Zookeeper connection string from step 1a and "InfosphereEvents" as the name of your Kafka topic.
Kafka message consumers are not limited to capturing messages from InfoSphere Information Server. You can also retrieve messages from your own applications when you implement your own Kafka producer.
Implement a Kafka producer for InfoSphere Information Server messages.
1. Design your own producer.
2. Run your producer. Use the Zookeeper connection string from step 1a and "InfosphereEvents" as the name of your Kafka topic.