Question & Answer
Question
How can I use Kafka to manage events that are created by InfoSphere Information Server?
Answer
You can use Apache Kafka to manage events created by InfoSphere Information Server. You can use this open source infrastructure to react to events as you wish, and create your own event source to connect custom systems.
Apache Kafka is an open-source publish-and-subscribe messaging system built for high throughput, speed, availability and scalability. InfoSphere Information Server has a ready-to-use installation of Kafka, version 0.8.2.1, and a Kafka topic that provides all Information Server events as Kafka messages. This allows you to see all messages from all suite products and to consume these messages with any of the existing Kafka client implementations.
For more information about Kafka, see https://kafka.apache.org/
Getting Started
Starting a command-line Kafka consumer and inspecting InfoSphere Information Server messages
Kafka comes with a command-line consumer that directs messages to a command window. You can use that consumer to see messages created by InfoSphere Information Server. Kafka uses Zookeeper, which is a centralized service for maintaining configuration information. You must determine the Zookeeper connection string before you can start the consumer.
1. Determine the Zookeeper connection string.
- 1. In a command window, change to the change to the following directory:
UNIX: <IS_Install_path>/ASBServer/bin
Windows: <IS_Install_path>\ASBServer\bin
2. Enter the following command (on UNIX, use the iisadmin.sh script)
UNIX: ./iisadmin.sh -display -key com.ibm.iis.sdp.zookeeper.connect
Windows: iisadmin -display -key com.ibm.iis.sdp.zookeeper.connect
The command returns the Zookeeper connection string in the form: hostname:port
- 1. Open an elevated command window and change to the following directory:
UNIX: <IS_install_path>/shared-open-source/kafka/install/bin
Windows: <IS_install_path>\shared-open-source\kafka\install\bin\windows
2. Enter the following command
UNIX: ./kafka-console-consumer.sh --zookeeper “hostname:port” --topic “InfosphereEvents” –from-beginning
Windows: kafka-console-consumer.bat --zookeeper “hostname:port” --topic “InfosphereEvents” –from-beginning
Note 'hostname:port' is the connection string you got in the previous steps. 'InfosphereEvents' is the name of the Kafka topic that is used by InfoSphere Information Server.
The console will print all InfoSphere Information Server messages.
For example, if you create an Information Governance Rule with InfoSphere Information Governance Catalog, the following message is created:
- {
“eventType”:”IGC_NEWRULE_EVENT”
“applicationType”:”Information Governance Catalog”
"ASSET_TYPE":"Information Governance Rule",
"ASSET_RID":"6662c0f2.e1b13efc.8gi55ct6c.l1eh0ds.3ttrq2.dueampgjs67s528k6b1el",
"ASSET_CONTEXT":"",
"ACTION":"CREATE",
"ASSET_NAME":"newrule"
}
Each message is a JSON object. If you use a Kafka client API to read messages from the InfosphereEvents topic, you can use JSON APIs to process the messages. If you choose to send your own messages to the InfosphereEvents Kafka topic, using the same message format as the above example will simplify your consumer implementation.
Implementing a Kafka consumer for InfoSphere Information Server messages
Kafka allows you to process InfoSphere Information Server messages to do things such as monitor your assets or trigger other systems, such as workflow engines or emails, from within your own code. Before you can process messages, you must implement a Kafka consumer.
Before you begin
Run the steps in the previous section to determine your Zookeeper connection string and become familiar with the InfoSphere Information Server message format.
1. Use this Apache Kafka consumer sample to design your own.
2. Run your example
- 1. Use the Zookeeper connection string you determined
2. Use 'InfosphereEvents' as the name of your Kafka topic.
Implementing a Kafka producer for InfoSphere Information Server messages
Kafka message consumers are not limited to capturing messages from InfoSphere Information Server. You can also retrieve messages from your own applications when you implement your own Kafka producer.
Before you begin
Run the steps in the previous section to determine the Zookeeper connection string and become familiar with the InfoSphere Information Server message format.
Use this Apache Kafka producer sample to design your own. Run the sample using the Zookeeper connection string you determined.
Related Information
Was this topic helpful?
Document Information
Modified date:
16 June 2018
UID
swg21977431