Configuring integration servers to send logs and events to Logstash in an ELK stack

You can configure your integration servers to send any or all of your logging and event data, in the form of BIP message logs, message flow monitoring events, and activity logging events, to a Logstash input plug-in in an Elasticsearch, Logstash, and Kibana (ELK) stack. This configuration enables you to display the reported information in a Kibana dashboard.

Complete the following steps to enable your integration servers to send logging and event data to a Logstash input in an ELK stack:

1. Decide which of the following Logstash input protocols that you want to use:
   • beats
   • beatsTls
   • http
   • https

   This information determines the property values that you set in the following steps.

   For more information about the Logstash input protocols that you can use to connect IBM App Connect Enterprise to the ELK stack, see the Logstash reference documentation online.

2. Open the node.conf.yaml or server.conf.yaml configuration file for your integration node or server, by using a YAML editor.

   You can edit the file by using the built-in YAML editor that is provided in the IBM App Connect Enterprise Toolkit, either by double-clicking the file in the Application Development view or by right-clicking the file and selecting Open with > YAML editor. If you choose to edit the file by using a plain text editor, ensure that you do not include any tab characters (which are not valid in YAML) and use a YAML validation tool to validate the contents of your file.

3. Configure the integration node or server to use your chosen Logstash input protocol (beats, beatsTls, http, or https), by setting properties in the ELKConnections section of the node.conf.yaml or server.conf.yaml file:

   
   ELKConnections:
     # Description for ELK Connections.
     # elkConnection1:
     #   elkProtocol: 'beats'                       # Logstash input protocol. Valid values are: 'beats', 'beatsTls', 'http', or 'https'.
     #   hostname: 'myhost.domain.com'              # Hostname for the elkProtocol endpoint.
     #   port: 0                                    # Port for the elkProtocol endpoint.
     #   uploadIntervalMilliSecs: 60000             # Interval between uploading cached data, set in milliseconds.
     #   elkCredential: ''                          # Set an 'elk' credential alias name to enable basic authentication, if it is required by the Logstash input protocol.
     #   keystoreFile: '/path/to/keystore.jks'      # Set the path to the keystore to be used, if it is required by the Logstash input protocol.
     #   keystorePass: 'P4s5w0rd'                   # Set the password, or 'keystore' credential alias to the password, of the keystore.
     #   keyAlias: ''                               # Set the alias name of the private key, if mutual authentication is required by the Logstash input protocol.
     #   KeyPassword: ''                                # Set the password, or 'keystorekey' credential alias to the password, for accessing the private mutual authentication key.
     #   truststoreFile: '/path/tp/truststore.jks'  # Set the path to the truststore to be used, if it is required by the Logstash input protocol.
     #   truststorePass: 'P4s5w0rd'                 # Set the password, or 'truststore' credential alias to the password, for accessing the truststore.

   For example, to connect to an unsecured Logstash input plug-in by using the beats input protocol, on localhost port 5044, and to upload the log data every 30 seconds, set the following properties:

   
   ELKConnections:
     elkbeats:
       elkProtocol: 'beats'
       hostname: 'localhost'
       port: 5044
       uploadIntervalMilliSecs: 30000 
   

4. Optional: If you are connecting to a secured Logstash input plug-in, configure the security credentials that are required to access it. For example, if you are using HTTP to connect to a Logstash input plug-in that requires a username and password, configure the integration node or server to use an elk credential for basic authentication (basicAuth), by completing the following steps:
   1. Update the node.conf.yaml or server.conf.yaml file to specify a credential to be used for basic authentication (basicAuth), by specifying the credential name in the elkCredential property; for example:

       elkCredential: 'elk_ID'   

   2. Use the mqsicredentials command to create the security credential that you specified in the .conf.yaml file (elk_ID), setting the username and password, and a credential type of elk:

      mqsicredentials --work-dir elk_work_dir --create --credential-name elk_ID --credential-type elk --username user1 --password passw0rd1

5. Optional: Enable App Connect Enterprise BIP messages to be sent to the configured ELK Connection, by setting properties in the Log section of the .conf.yaml file:

   
   Log:
     #consoleLog: true          # Control writing BIP messages to standard out. Set to true or false, default is true.
     #outputFormat: 'text'      # Control the format of BIP messages written to standard out and file. Set to ibmjson or text, default is text.
     #eventLog: '[iib.system-work-dir]/log/[iib.system-node-label].[iib.system-server-label].events.txt'     # Control writing BIP messages to file. Set to '' to disable, default is as shown.
     #eventLogFileSize: 10      # The maximum size in MB of an event log file before it is rotated into a new file
     #eventLogFileCount: 10     # The maximum number of event log files that should be rotated between.
     #elkLog: false             # Control the publication of BIP messages to an ELK (Elasticsearch, Logstash, Kibana) stack. Set to true or false, default is false.
     #elkConnections: ''        # Name of the ELK connection to use, for example 'elkConnection1'
                                # Each named ELK Connection must be defined in the ELKConnections section below.
   

   For example, to enable logs data to be sent to the ELK Connection that was configured in the previous example (elkbeats), set the following properties:

   
   Log:
     elkLog: true
     elkConnections: 'elkbeats'

6. Optional: Optionally, enable IBM App Connect Enterprise message flow monitoring events to be sent to the configured ELK Connection, by setting properties in the Events.BusinessEvents.ELK section of the .conf.yaml file:

   
   Events:
     BusinessEvents:
       ELK:
         #enabled: false               # Set true or false, default false
         #outputFormat: 'json'      # Set json, default json
         #elkConnections: ''           # Name of the ELK connection to use, for example 'elkConnection1', must be defined in the ELKConnections section below.

   Also, ensure that Monitoring is enabled in the Monitoring.MessageFlow section of the .conf.yaml file:

   
   Monitoring:
     MessageFlow:
       #publicationOn: 'inactive'         # choose 1 of : active|inactive, default inactive

7. Optional: Deploy an Activity Log policy to the integration server, with the elkLog property set to true and the elkConnections property set to match the name of an ELK connection configuration specified in ELKConnections section of the .conf.yaml file.
8. Restart the integration server for the changes to take effect.
   The integration server's log events in the form of BIP message logs, message flow monitoring events (or both), are sent to the configured Logstash input plug-in.

   When the integration server starts delivering BIP message logs, message flow monitoring events (or both), to the configured Logstash input, a confirmation message is written to the log; for example:

   BIP6503I: ( CHECK.IS2 ) The integration server successfully sent data to ELK connection ''ELKbeats'' 
   using elkProtocol ''beats'', hostname ''localhost'' and port ''5444''.

   If the integration server is unable to deliver event data to the configured Logstash input, a BIP message is logged to the integration server's local event log; for example:

   BIP3888E: ( CHECK.IS2 ) The ELK connector ''ELKbeats'' failed to send data to ''localhost:5444''. 
   Error details: ''SocketException BIP3150E: ImbBasicSocket::connectTimeout "An error occurred whilst performing a socket operation: 
   getsockopt" [::connect::select(), 10061, No connection could be made because the target machine actively refused it.