Configuring Logstash in ICP from the command line

Procedure for configuring Logstash to point to the probe's webhook running on ICP. It can also be used with the on-premises version of the probe.

Modifying Logstash configuration on IBM Cloud Private 3.2.0

To modify the default Logstash configuration, use the following steps:

Determine the Logstash Pipeline ConfigMap in the same namespace. In this procedure, the ConfigMap in the kube-system namespace is logging-elk-logstash-pipeline-config. If a separate Logstash is deployed, determine the pipeline ConfigMap and add a new http output. Note: In ICP 3.1.2 or below, the Logstash Pipeline ConfigMap name is logging-elk-logstash-config.
Edit the Logstash pipeline ConfigMap to add a new http output. To do this via the command line, configure kubectl client and follow the steps below.
Load the config map into a file using the following command:
kubectl get configmap logging-elk-logstash-pipeline-config --namespace=kube-system -o yaml > logging-elk-logstash-pipeline-config.yaml
Edit the logging-elk-logstash-pipeline-config.yaml file and modify the output object to add a new http output object as shown below (using the full webhook URL as shown in step 1 above in the http.url parameter):
```
output {
      elasticsearch {
        index => "logstash-%{+YYYY.MM.dd}"
        hosts => "elasticsearch:9200"
      }
       http {
         url => "http://<ip_address>:<port>/probe/webhook/logstash"
         format => "json"
         http_method => "post"
         pool_max_per_route => "5"
       }
    }
```
Note: (Optional) The pool_max_per_route is set to limit concurrent connections to the probe to 5 so that Logstash does not flood the probe which may cause event loss.

Save the changes in the file and replace the config map using the following command:

kubectl replace --namespace kube-system logging-elk-logstash-pipeline-config -f logging-elk-logstash-pipeline-config.yaml
configmap "logging-elk-logstash-pipeline-config" replaced

Note: It usually takes a minute or so for Logstash to reload the new configmap.

Check the logs to make sure there are no errors sending HTTP POST notifications to the probe.