Preparing to send data to Humio via Logstash

To send data from Z Common Data Provider to Humio via Logstash, configure Logstash by using the Logstash configuration files that are provided by Z Common Data Provider.

In preparation for sending data to Humio, complete the following steps:

1. Log in to the Logstash server, and extract the Z Operational Log and Data Analytics Elastic Stack ingestion kit for raw data, which is in the file ZLDA-IngestionKit-raw-v.r.m.f.zip. By default, the files are extracted into the zlda-config-raw directory. For more information about how to get the package, see Obtaining and preparing the installation files.
2. Extract the Elasticsearch ingestion kit to access the Logstash configuration files.
3. Create a new directory under the Logstash installation directory and copy the Logstash configuration files that you need for your environment to the new directory.
   The column Prefix in file name of Logstash configuration file in the following table indicates the prefixes that are used in the file names for the Logstash configuration files in the Z Common Data Provider Elasticsearch ingestion kit. The file name prefix is an indication of the configuration file content.

   Table 1. Mapping of the prefix that is used in a Logstash configuration file name to the content of the file

   Prefix in file name of Logstash configuration file	Content of configuration file with this prefix
   B_	Input stage
   E_	Preparation stage
   H_	Field name annotation stage
   N_	Timestamp resolution stage
   Q_	Output stage

   The following descriptions further explain the Logstash configuration files in the Z Common Data Provider Elasticsearch ingestion kit:

   B_CDPz_Input.conf file

   This file contains the input stage that specifies the TCP/IP port on which Logstash listens for data from the Data Streamer. Copy this file to your Logstash configuration directory. You can specify the following input and filter.

   input { 
           tcp { 
                   port => 8080 
           }
   }

   E_CDPz_Index.conf file

   This file contains the preparation stage. Copy this file to your Logstash configuration directory.

   Files with H_ prefix in file name

   Each of these files contains a unique field name annotation stage that maps to a unique data stream that Z Common Data Provider can send to Logstash. To your Logstash configuration directory, copy the H_ files for only the data streams that you want to send to Elasticsearch.

   Files with N_ prefix in file name

   Each of these files contains a unique timestamp resolution stage that maps to a unique data stream that Z Common Data Provider can send to Logstash. To your Logstash configuration directory, copy the N_ files for only the data streams that you want to send to Elasticsearch.

   Q_CDPz_Elastic.conf file

   This file contains an output stage that sends all records to a single Elasticsearch server. Copy this file to your Logstash configuration directory.

   After you copy the file, edit it according to the following example:

   output {
   	elasticsearch {
       hosts => [ "humio_url/api/v1/ingest/elastic-bulk" ]   
   		user => "humio_user"
   		password => "ingest_token"
   	}
   }

   humio_url

   The Humio server URL, for example, http://localhost:8080.

   humio_user

   The Humio user.

   ingest_token

   The Humio repository token. If you don't already have one, go to the Setting tab on the Humio repository UI and click API Tokens to enter the token configuration interface. Then click the Copy button in the selected token column in the Tokens section.

4. In the script for starting Logstash, specify the directory that you created in step 3.
5. In the Configuration Tool, define a policy with one of the following protocols.
   • Logstash
   • Logstash secure
6. Start Logstash and Humio.
   If the activation is successful, Z Common Data Provider starts sending data to Humio.