Configuring integration servers to send logging data to Logstash in an ELK stack
You can configure your integration servers to send logging data (event log BIP messages) to a Logstash input plug-in in an Elasticsearch, Logstash, and Kibana (ELK) stack, so that you can display the reported information in a Kibana dashboard.
Before you begin
About this task
To enable your IBM App Connect Enterprise integration servers to send logging information (BIP messages) to a Logstash input in an ELK stack, you must configure the integration node or server by setting the properties in the node.conf.yaml or server.conf.yaml file.
For more information about configuring an integration node or server, see Configuring an integration node by modifying the node.conf.yaml file or Configuring an integration server by modifying the server.conf.yaml file.
Complete the following steps to enable your integration servers to send logging data to a Logstack input in an ELK stack:
- Decide which of the following Logstash input protocols you will use:
For more information about the Logstash input protocols that you can use to connect IBM App Connect Enterprise to the ELK stack, see the Logstash reference documentation online.
- Open the node.conf.yaml or server.conf.yaml
configuration file for your integration node or server, by using a YAML editor.
If you are not able to access a YAML editor, you can edit the file by using a plain text editor. However, you must ensure that you do not include any tab characters. Tab characters are not valid in YAML files and they would cause your configuration to fail. If you are using a plain text editor, ensure that you use a YAML validation tool to validate the content of your file.
- Configure the integration node or server to use your chosen Logstash input protocol
https), by setting properties in the
ELKConnectionssection of the node.conf.yaml or server.conf.yaml file:
ELKConnections: # Description for ELK Connections. # elkConnection1: # elkProtocol: 'beats' # Logstash input protocol. Valid values are: 'beats', 'beatsTls', 'http', or 'https'. # hostname: 'myhost.domain.com' # Hostname for the elkProtocol endpoint. # port: 0 # Port for the elkProtocol endpoint. # uploadIntervalMilliSecs: 60000 # Interval between uploading cached data, set in milliseconds. # elkCredential: '' # Set an 'elk' credential alias name to enable basic authentication, if it is required by the Logstash input protocol. # keystoreFile: '/path/to/keystore.jks' # Set the path to the keystore to be used, if it is required by the Logstash input protocol. # keystorePass: 'P4s5w0rd' # Set the password, or 'keystore' credential alias to the password, of the keystore. # keyAlias: '' # Set the alias name of the private key, if mutual authentication is required by the Logstash input protocol. # KeyPassword: '' # Set the password, or 'keystorekey' credential alias to the password, for accessing the private mutual authentication key. # truststoreFile: '/path/tp/truststore.jks' # Set the path to the truststore to be used, if it is required by the Logstash input protocol. # truststorePass: 'P4s5w0rd' # Set the password, or 'truststore' credential alias to the password, for accessing the truststore.For example, to connect to an unsecured Logstash input plug-in by using the
beatsinput protocol, on
5044, and to upload the log data every 30 seconds, set the following properties:
ELKConnections: elkbeats: elkProtocol: 'beats' hostname: 'localhost' port: 5044 uploadIntervalMilliSecs: 30000
- Optional: If you are connecting to a secured Logstash input plug-in,
configure the security credentials that are required to access it. For example, if you are using
HTTP to connect to a Logstash input plug-in that requires a username and password, configure the
integration node or server to use an
elkcredential for basic authentication (basicAuth), by completing the following steps:
- Update the
server.conf.yamlfile to specify a credential to be used for basic authentication (basicAuth), by specifying the credential name in the elkCredential property; for example:
- Use the mqsicredentials command to create the security credential that you
specified in the .conf.yaml file (
elk_ID), setting the username and password, and a credential type of
mqsicredentials --work-dir elk_work_dir --create --credential-name elk_ID --credential-type elk --username user1 --password passw0rd1
- Update the
- Enable App Connect Enterprise BIP messages to be sent to the configured
ELK Connection, by setting properties in the
Logsection of the .conf.yaml file:
Log: #consoleLog: true # Control writing BIP messages to standard out. Set to true or false, default is true. #outputFormat: 'text' # Control the format of BIP messages written to standard out and file. Set to ibmjson or text, default is text. #eventLog: '[iib.system-work-dir]/log/[iib.system-node-label].[iib.system-server-label].events.txt' # Control writing BIP messages to file. Set to '' to disable, default is as shown. #eventLogFileSize: 10 # The maximum size in MB of an event log file before it is rotated into a new file #eventLogFileCount: 10 # The maximum number of event log files that should be rotated between. #elkLog: false # Control the publication of BIP messages to an ELK (Elasticsearch, Logstash, Kibana) stack. Set to true or false, default is false. #elkConnections: '' # Name of the ELK connection to use, for example 'elkConnection1' # Each named ELK Connection must be defined in the ELKConnections section below.For example, to enable BIP messages to be sent to the ELK Connection that was configured in the previous example (
elkbeats), set the following properties:
Log: elkLog: true elkConnections: 'elkbeats'
- Restart the integration server for the changes to take effect. The integration server's log events (BIP messages) will now be sent to the configured Logstash input plug-in.When the integration server starts delivering BIP messages to the configured Logstash input, a confirmation message is written to the log; for example:
BIP6503I: ( CHECK.IS2 ) The integration server successfully sent data to ELK connection ''ELKbeats'' using elkProtocol ''beats'', hostname ''localhost'' and port ''5444''.If the integration server is unable to deliver event data to the configured Logstash input, a BIP message is logged to the integration server's local event log; for example:
BIP3888E: ( CHECK.IS2 ) The ELK connector ''ELKbeats'' failed to send data to ''localhost:5444''. Error details: ''SocketException BIP3150E: ImbBasicSocket::connectTimeout "An error occurred whilst performing a socket operation: getsockopt" [::connect::select(), 10061, No connection could be made because the target machine actively refused it.