Configuring Insight Packs that use LFA to load data and Logstash to annotate it

To integrate the scalable data collection architecture with any Insight® Packs that use the LFA and Logstash to stream data, annotate it and send it to Log Analysis, you need to adapt the configuration to make it compatible with scalable data collection.

Before you begin

Configure the scalable data collection architecture. For more information, see Configuring scalable data collection.

About this task

This configuration is intended for Insight Packs that use the LFA to stream data to Logstash where it is processed and sent to Log Analysis. For example, the Generic Receiver Insight Pack, which is part of the Log Analysis health pack.

You copy the Logstash configurations that are included in the Insight Packs to the Sender and Receiver cluster configurations in your scalable data collection architecture. First, copy the input section from the Logstash configuration file for the Insight Pack to the Receiver configuration file. You also need to add some configuration for processing the metadata fields and sending messages to the topics and partitions in Apache Kafka. Next, you copy the filtering and processing logic from the Logstash configuration for the Insight Pack to the Sender cluster configuration. These configuration settings help process the log files and forward them to Log Analysis.

Procedure

Update the LFA configuration for the Insight Pack so that it can send data to HAProxy or the Receiver cluster.
Add the server and port information for the Receiver cluster instance or the HAProxy to the LFA configuration or .conf file. For example:
```
ServerLocation=<HAProxy_or_receiver_cluster_server>
ServerPort=<HAProxy_or_receiver_cluster_port>
```
For more information, see Configuring the Log File Agent.
Configure the tcp input section of the Receiver cluster configurations so that it can receive data that is sent by Logstash.
For example:
```
input {
        tcp {
                port => <Logstash_Port>
                type => LFA
                codec => line { charset => US-ASCII }
        }
}
```
For more information, see Configuring the Receiver cluster for single line logs.

Update the Receiver cluster to process data and send it to the Apache Kafka brokers.

To update the Receiver cluster:

Copy the matching patterns from the Insight Pack configuration to the Logstash servers in the Receiver cluster.
Update the input section with the input section from the Logstash configuration for the Insight Pack.

Update the filter section with the matching logic from the Logstash configuration files for the Insight Pack and add fields that are mapped to the Apache Kafka topic or partition. For example:


filter {
   if [type] == LFA{
       mutate {
           strip => [message]
       }
       grok {
           match => [ message, %{LFAMESSAGE} ]
           patterns_dir => [<Patterns_directory>]
           add_tag => [grok_lfa_prod]
       } # end LFA grok
       if grok_lfa_prod not in [tags]{
           grok{
               match => [ message, %{LALFAMESSAGE} ]
               patterns_dir => [<Patterns_directory>]
               add_tag => [grok_lfa]
           }
       }
   }

   if ( grok_lfa_prod in [tags] or grok_lfa in [tags] ) {
      mutate {
         add_field => [ datasource, LA_Health_Pack ]
         add_field => [ resourceID, %{LFA_HOSTNAME}_%{LFA_LOGNAME} ]
      }
   }
}

This example shows the output section:


output {
        if (grok_lfa in [tags]) and ! (_grokparsefailure in [tags]) {
            kafka {
                bootstrap_servers =>
 <Kafka_broker_server1>:<kafka_broker_port1>,..
                topic_id => %{datasource}
                message_key => %{resourceID}
            }
        }
}

The datasource field is mapped to a topic in Apache Kafka. The reasourceID is mapped to a partition.

Copy the remaining portions of the Logstash configuration file for the Insight Pack to the Sender configuration file.
Update the Sender configuration file so that it can read data from the topics and partitions in Apache Kafka and send it to Log Analysis.
To update the Sender configuration:
1. Copy the matching patterns from the Logstash configuration file for the Insight Pack to the Logstash servers in the Sender cluster.
2. Update the Input section of the Sender cluster configuration to read data from the topic or partition in Apache Kafka. For example:
```
input {
        kafka {
                zk_connect => <Zookeeper_Host>:<Zookeeper_Port>
                group_id => LA_Health_Pack
                topic_id => LA_Health_Pack
                consumer_threads => 4
                consumer_restart_on_error => true
                consumer_restart_sleep_ms => 100
                decorate_events => true
        }

} #end inputs
```
3. Update the filter section of the Sender configuration with the remaining configuration from the Logstash configuration file for the Insight Pack.
4. Update the Output section to send data to Log Analysis.
  For more information, see Streaming data with Logstash.