Integrating Logstash with Netcool Operations Insight

To modify the default Logstash configuration, use the following steps:

  1. Deploy the ibm-netcool-probe chart.
  2. After a successful deployment, get the Logstash probe's Endpoint Host and Port from the Workloads > Deployments page.
    • If logstashPobe.service.type is set to ClusterIP, the full webhook URL will have the following format: http://<service name>.<namespace>:<externalPort>/probe/webhook/logstash

      To obtain the service name and port using the command line, use the following commands substituting <namespace> with the namespace where the release is deployed and <release_name> with the Helm release name.

      # Get the Service name export SVC_NAME=$(kubectl get services --namespace <namespace> -l "app.kubernetes.io/instance=<release_name>,app.kubernetes.io/component=logstashprobe" -o jsonpath="{.items[0].metadata.name}")

      # Get the Service port number export SVC_PORT=$(kubectl get services --namespace <namespace> -l "app.kubernetes.io/instance=<release_name>,app.kubernetes.io/component=logstashprobe" -o jsonpath="{.items[0].spec.ports[0].port}")

    • If logstashPobe.service.type is set to Nodeport, the full webhook URL will have the following format: http://<External IP>:<Node Port>/probe/webhook/logstash

      To obtain the NodePort number using the command line, use the following commands substituting <namespace> with the namespace where the release is deployed and <release_name> with the Helm release name.

      # Get the NodePort number from the Service resource export NODE_PORT_LOGSTASH=$(kubectl get services --namespace <namespace> -l "app.kubernetes.io/instance=<release_name>,app.kubernetes.io/component=logstashprobe" -o jsonpath="{.items[0].spec.ports[0].nodePort}")

      # On ICP 3.1.1, you can obtain the External IP from the IBM Cloud Cluster Info Configmap using the command below. export NODE_IP_LOGSTASH=$(kubectl get configmap --namespace kube-public ibmcloud-cluster-info -o jsonpath="{.data.proxy_address}")

      # On ICP 3.1.0, get the External IP from the Nodes resource. This command requires Cluster Administrator role. export NODE_IP_LOGSTASH=$(kubectl get nodes -l proxy=true -o jsonpath="{.items[0].status.addresses[0].address}")

  3. Determine the Logstash Pipeline config map in the same namespace. In this procedure, the ConfigMap in the kube-system namespace is logging-elk-logstash-config. If a separate Logstash is deployed, determine the pipeline ConfigMap and add a new http output.
  4. Edit the Logstash pipeline ConfigMap to add a new http output. To do this using the command line, configure the kubectl client and follow the steps below.
  5. Load the ConfigMap into a file using the following command:

    kubectl get configmap logging-elk-logstash-config --namespace=kube-system -o yaml > logging-elk-logstash-config.yaml

  6. Edit the logging-elk-logstash-config.yaml file. Modify the output object to add a new http output object as shown below. Use the full webhook URL as shown in Step 2 in the http.url parameter.
    output {
      elasticsearch {
        index => "logstash-%{+YYYY.MM.dd}"
        hosts => "elasticsearch:9200"
      }
       http {
         url => "http://<ip_address>:<port>/probe/webhook/logstash"
         format => "json"
         http_method => "post"
         pool_max_per_route => "5"
       }
    }
    Note: (Optional) pool_max_per_route is set to limit concurrent connections to the probe to 5 so that Logstash does not flood the probe which may cause event loss.
  7. Save the changes in the file and replace the ConfigMap. 
    kubectl replace --namespace kube-system logging-elk-logstash-config -f logging-elk-logstash-config.yaml

configmap "logging-elk-logstash-config" replaced
  8. Logstash takes a minute or so to reload the new configration. Check the logs to make sure there are no errors sending HTTP POST notifications to the probe.