Preparing to send data to Apache Kafka

If you use Apache Kafka as a target destination, you must install and configure it on your own or as part of a solution package that includes Kafka, for example, the Z Data Analytics platform. This procedure summarizes how to configure Kafka if you have installed it on your own.

Before you begin

Note: To set up Apache Kafka on IBM Z®, see Setting up Apache Kafka for IBM Z.

About this task

If you are using the Log Forwarder or the System Data Engine as the data gatherer, you must either create the header topic and data topics on Apache Kafka manually, or set auto.create.topics.enable=true in the server.properties file to enable auto creation of topics on Apache Kafka. See the Apache Kafka official website for more information.
Note: Parameter auto.create.topics.enable is set to true by default, which means Apache Kafka will automatically create topics unless you turn off this capability.

Procedure

If you use other data gatherers other than the Data Collector and set the parameter auto.create.topics.enable to false, you need to complete the following steps.

  1. Create the header topic named as prefix-CSV-HEADER.
    • Set cleanup.policy=compact for the header topic.
    • The default prefix is IBM-CDP.
    • You can use a customized prefix. In the Configuration Tool, set the Enable Customized Kafka Topic field to Yes and customize the value of the Prefix field for the subscriber.
  2. Create a data topic or data topics depending on whether customized topic is used.
    • If customized topic is used,
      • If you have not created a policy, create a data topic whose name is prefix-customizedtopic, for example: IBM-CDP-CUSTOM.
      • If you have created a policy in the Configuration Tool, you can click the Configure icon Configure icon on the subscriber node in your policy, and use the topic that you have customized for the Customized Topic field.
    • If customized topic is not used, create data topics for all data source types that you want to collect. The topic names are IBM-CDP-function-datasourcetype, for example: IBM-CDP-RAW-zOS-SMF_080. To get the default Kafka topic names of different data streams, refer to the Default Kafka Topic field on the data stream node in your policy from the Configuration Tool.