Preparing to send data to Elastic Stack

To send data from Z Common Data Provider to Elasticsearch, configure Logstash by using the Logstash configuration files that are provided by Z Common Data Provider.

In preparation for sending data to Elasticsearch, complete the following steps:

1. From the product's or suite's installation directory, download the Elasticsearch ingestion kit, which is in the ibm_cdpz_ELK.tar.gz file, in binary mode.
2. Extract the Elasticsearch ingestion kit to access the Logstash configuration files.
3. Create a directory under the Logstash installation directory and copy the Logstash configuration files that you need for your environment to the new directory.
   Table 1 indicates the prefixes that are used in the file names for the Logstash configuration files in the Z Common Data Provider Elasticsearch ingestion kit. The file name prefix is an indication of the configuration file content.

   Table 1. Mapping of the prefix that is used in a Logstash configuration file name to the content of the file

   Prefix in file name of Logstash configuration file	Content of configuration file with this prefix
   B_	Input stage
   E_	Preparation stage
   H_	Field name annotation stage
   N_	Timestamp resolution stage
   Q_	Output stage

   The following descriptions further explain the Logstash configuration files in the Z Common Data Provider Elasticsearch ingestion kit:

   B_CDPz_Input.lsh file

   This file contains the input stage that specifies the TCP/IP port on which Logstash listens for data from the Data Streamer. Copy this file to your Logstash configuration directory. You might need to edit the port number after you copy the file.

   B_CDPz_Kafka.lsh file

   This file is used only when you stream non-OMEGAMON® data from Apache Kafka to Logstash, for example, if you stream non-OMEGAMON data from the Z Data Analytics Platform to Logstash, you need to use this file.

   It contains the input stage that specifies the bootstrap server on which Logstash listens for data from Apache Kafka. Update the bootstrap server in this file as appropriate for your environment.

   Note: You can only have one input stage in your configuration. If you use B_CDPz_Kafka.lsh, you must remove B_CDPz_Input.lsh and B_CDPz_Omegamon.lsh.

   E_CDPz_Index.lsh file

   This file contains the preparation stage. Copy this file to your Logstash configuration directory.

   Files with H_ prefix in file name

   Each of these files contains a unique field name annotation stage that maps to a unique data stream that Z Common Data Provider can send to Logstash. To your Logstash configuration directory, copy the H_ files for only the data streams that you want to send to Elasticsearch.

   Files with N_ prefix in file name

   Each of these files contains a unique timestamp resolution stage that maps to a unique data stream that Z Common Data Provider can send to Logstash. To your Logstash configuration directory, copy the N_ files for only the data streams that you want to send to Elasticsearch.

   Q_CDPz_Elastic.lsh file

   This file contains an output stage that sends all records to a single Elasticsearch server. Copy this file to your Logstash configuration directory.

   After you copy the file, edit it to add the name of the host to which the stage is sending the indexing call. The default name is localhost, which indexes the data on the server that is running the ingestion processing. Change the value of the hosts parameter rather than the value of the index parameter. The index value is assigned during ingestion so that the data for each source type is sent to a different index. The host determines the Elasticsearch farm in which the data is indexed. The index determines the index in which the data is held.

   To split data according to sysplex, you can use the [sysplex] field in an if statement that surrounds an appropriate Elasticsearch output stage.

   The following files are used only when you stream OMEGAMON data to Logstash. Copy the following configuration files from the ingestion kit to your Logstash configuration directory. For more information about how to configure the files, see Streaming OMEGAMON data from Kafka to the Elastic Stack.

   B_CDPz_Omegamon.lsh file

   It contains the input stage that specifies the TCP/IP port on which Logstash listens for data from the Data Streamer. Specify the port on which Logstash listens for data from the Data Streamer. The default value is 8080.

   Note: You can only have one input stage in your configuration. If you use B_CDPz_Omegamon.lsh, you must remove B_CDPz_Input.lsh and B_CDPz_Kafka.lsh.

   CDPz_Omegamon.lsh file

   This file contains the information of how Logstash parses and splits the concatenated JSON data, and a unique field name annotation stage that maps to OMEGAMON data.

   Q_CDPz_Omegamon.lsh file

   It contains an output stage that sends all records to a single Elasticsearch server. Copy this file to your Logstash configuration directory.

   After you copy the file, edit it to change the value of the hosts parameter to the IP address where Elasticsearch is running. The default value is localhost.

4. In the script for starting Logstash, specify the directory that you created in step 3.
5. Define a policy with the Logstash as the subscriber.
6. Start Logstash and Elasticsearch.
   If the activation is successful, Z Common Data Provider starts sending data to Elasticsearch.