Reporting logging data to Logstash in an ELK stack
IBM® App Connect Enterprise provides a facility to send BIP messages for your integration servers to a Logstash input in an Elasticsearch, Logstash, and Kibana (ELK) stack, so that you can view that data in a Kibana dashboard.
- Elasticsearch (a search and analytics engine)
- Logstash (a data processing pipeline that ingests data and sends it to a stash, such as Elasticsearch)
- Kibana (a visualization tool that can be used to display data in charts and graphs).
BIP messages that are generated by IBM App Connect Enterprise integration servers (not integration nodes) can be sent to Logstash. However, you can enable logging at the level of an integration server or an integration node, by setting properties in the server.conf.yaml or node.conf.yaml file. To configure a specific integration server to send BIP messages to Logstash, you set logging properties in the server.conf.yaml file for that integration server. If you enable logging for an integration node (by setting logging properties in the node.conf.yaml file), events that are generated by all of its managed integration servers will be sent to Logstash.
When the log reporting feature is active, logging data (in the form of BIP messages) is sent to
Logstash at regular intervals, which you can specify in the .conf.yaml file.
You can also specify the Logstash input protocol to be used for sending the data
(beats, beatsTls, http, or
https). For information about how to configure your integration servers to report
logging data to Logstash, see Configuring integration servers to send logging data to Logstash in an ELK stack.
When you activate the IBM App Connect Enterprise logging capability and specify the Logstash input protocol, BIP messages that are triggered by integration server events are sent to the Logstash input plug-in. BIP messages can be published from independent integration servers and from integration servers that are managed by an integration node. The logging data that is sent to Logstash contains information about the events that are issued by the integration server process; events that are initiated by other components (such as an integration node) are not reported. If you compare the contents of the local event log relating to integration servers with the logging data that is published to Logstash, you will see that a very small number of BIP messages at the beginning and end of the local log are not published to Logstash. These are typically messages about the startup and shutdown of the integration server. Publication of BIP messages to Logstash begins after the integration server has started, which means that messages relating to the integration server startup are not published to Logstash. BIP messages are then published until the integration server shutdown process begins, so any messages relating to the shut down are written to the local system log but are not published to Logstash. If an integration server does not appear to be delivering messages to Logstash, check the local system log for information. For more information about Logstash, see the Logstash reference documentation online.
If you want to send data to a secured Logstash input, you can configure the security credentials, including the username, password, truststore, and keystore, by setting properties in the integration node's node.conf.yaml file or the integration server's server.conf.yaml file. You can store the security credentials in the encrypted vault by using the mqsicredentials command and the mqsivault command. For information about encrypted security credentials, see Configuring encrypted security credentials.
For more information about configuring an integration server, see Configuring an integration server by modifying the server.conf.yaml file. For more information about viewing data in a Kibana dashboard, see the Elastic stack online documentation.