Configuring Logstash in ICP from the command line

Procedure for configuring Logstash to point to the probe's webhook running on ICP. It can also be used with the on-premises version of the probe.

Modifying Logstash configuration on IBM Cloud Private 3.2.0

To modify the default Logstash configuration, use the following steps:

  1. Determine the Logstash Pipeline ConfigMap in the same namespace. In this procedure, the ConfigMap in the kube-system namespace is logging-elk-logstash-pipeline-config. If a separate Logstash is deployed, determine the pipeline ConfigMap and add a new http output. Note: In ICP 3.1.2 or below, the Logstash Pipeline ConfigMap name is logging-elk-logstash-config.
  2. Edit the Logstash pipeline ConfigMap to add a new http output. To do this via the command line, configure kubectl client and follow the steps below.
  3. Load the config map into a file using the following command:

    kubectl get configmap logging-elk-logstash-pipeline-config --namespace=kube-system -o yaml > logging-elk-logstash-pipeline-config.yaml

  4. Edit the logging-elk-logstash-pipeline-config.yaml file and modify the output object to add a new http output object as shown below (using the full webhook URL as shown in step 1 above in the http.url parameter):
    output {
          elasticsearch {
            index => "logstash-%{+YYYY.MM.dd}"
            hosts => "elasticsearch:9200"
           http {
             url => "http://<ip_address>:<port>/probe/webhook/logstash"
             format => "json"
             http_method => "post"
             pool_max_per_route => "5"
    Note: (Optional) The pool_max_per_route is set to limit concurrent connections to the probe to 5 so that Logstash does not flood the probe which may cause event loss.
  5. Save the changes in the file and replace the config map using the following command:
    kubectl replace --namespace kube-system logging-elk-logstash-pipeline-config -f logging-elk-logstash-pipeline-config.yaml
    configmap "logging-elk-logstash-pipeline-config" replaced
    Note: It usually takes a minute or so for Logstash to reload the new configmap.
  6. Check the logs to make sure there are no errors sending HTTP POST notifications to the probe.