Reporting logs and monitoring events to a Logstash input in an ELK stack

IBM® App Connect Enterprise provides a facility to send any or all of your BIP message logs, message flow monitoring events, and activity logging events for your integration servers to a Logstash input in an Elasticsearch, Logstash, and Kibana (ELK) stack, so that you can view that data in a Kibana dashboard.

The ELK stack consists of three parts:
  • Elasticsearch (a search and analytics engine)
  • Logstash (a data processing pipeline that ingests data and sends it to a stash, such as Elasticsearch)
  • Kibana (a visualization tool that can be used to display data in charts and graphs).
For more information about Elasticsearch, Logstash, and Kibana, see the Elastic stack web pages.

BIP message logs, message flow monitoring events, and activity logging events that are generated by IBM App Connect Enterprise integration servers (not integration nodes) can be sent to Logstash. However, you can enable logging at the level of an integration server or an integration node, by setting properties in the server.conf.yaml or node.conf.yaml file. To configure a specific integration server to send logs and events to Logstash, you set logging properties in the server.conf.yaml file for that integration server. If you enable logging for an integration node (by setting logging properties in the node.conf.yaml file), events that are generated by all of its managed integration servers are sent to Logstash.

App Connect Enterprise requires the ELK stack to include Logstash with either beats or http input plug-in configured. When you configure App Connect Enterprise to send to Logstash, you select the beats or http protocol and specify connection and security parameters. When the reporting feature is active, logging and event data are sent to Logstash at regular intervals, which you can specify in the .conf.yaml file. You can also specify the Logstash input protocol to be used for sending the data (beats, beatsTls, http, or https). For more information about how to configure your integration servers to report logging and event data to Logstash, see Configuring integration servers to send logs and events to Logstash in an ELK stack.

When you activate the IBM App Connect Enterprise logging capability and specify the Logstash input protocol, BIP message logs, message flow monitoring events, and activity logging events that are triggered by integration server events are sent to the Logstash input plug-in. The logs and events can be published from independent integration servers and from integration servers that are managed by an integration node. The logging data that is sent to Logstash contains information about the events that are issued by the integration server process. Events that are initiated by other components (such as an integration node) are not reported. If you compare the contents of the local event log relating to integration servers with the logging data that is published to Logstash, you see that a very small number of BIP message logs, message flow monitoring events (or both) at the beginning and end of the local log are not published to Logstash. These logs and events are typically messages about the startup and shutdown of the integration server. Publication of logs and events to Logstash begins after the integration server starts, which means that messages that relate to the integration server startup are not published to Logstash. The logs and events are then published until the integration server shutdown process begins, so any messages that relate to the shutdown are written to the local system log but are not published to Logstash. If an integration server does not appear to be delivering messages to Logstash, check the local system log for information. For more information about Logstash, see the Logstash reference documentation online.

  • If you want to send data to a secured Logstash input, you can configure the security credentials, including the username, password, truststore, and keystore, by setting properties in the integration node's node.conf.yaml file or the integration server's server.conf.yaml file
. You can store the security credentials in the encrypted vault by using the mqsicredentials command and the mqsivault command. For more information about encrypted security credentials, see Configuring encrypted security credentials.

For more information about configuring an integration server, see Configuring an integration server by modifying the server.conf.yaml file. For more information about viewing data in a Kibana dashboard, see the Elastic stack online documentation.