Configuring integration servers to send logging data to Logstash in an ELK stack

You can configure your integration servers to send logging data (event log BIP messages) to a Logstash input plug-in in an Elasticsearch, Logstash, and Kibana (ELK) stack, so that you can display the reported information in a Kibana dashboard.

Complete the following steps to enable your integration servers to send logging data to a Logstack input in an ELK stack:

1. Decide which of the following Logstash input protocols you will use:
   • beats
   • beatsTls
   • http
   • https

   This information determines the property values that you will set in the following steps.

   For more information about the Logstash input protocols that you can use to connect IBM App Connect Enterprise to the ELK stack, see the Logstash reference documentation online.

2. Open the node.conf.yaml or server.conf.yaml configuration file for your integration node or server, by using a YAML editor.

   If you are not able to access a YAML editor, you can edit the file by using a plain text editor. However, you must ensure that you do not include any tab characters. Tab characters are not valid in YAML files and they would cause your configuration to fail. If you are using a plain text editor, ensure that you use a YAML validation tool to validate the content of your file.

3. Configure the integration node or server to use your chosen Logstash input protocol (beats, beatsTls, http, or https), by setting properties in the ELKConnections section of the node.conf.yaml or server.conf.yaml file:

   
   ELKConnections:
     # Description for ELK Connections.
     # elkConnection1:
     #   elkProtocol: 'beats'                       # Logstash input protocol. Valid values are: 'beats', 'beatsTls', 'http', or 'https'.
     #   hostname: 'myhost.domain.com'              # Hostname for the elkProtocol endpoint.
     #   port: 0                                    # Port for the elkProtocol endpoint.
     #   uploadIntervalMilliSecs: 60000             # Interval between uploading cached data, set in milliseconds.
     #   elkCredential: ''                          # Set an 'elk' credential alias name to enable basic authentication, if it is required by the Logstash input protocol.
     #   keystoreFile: '/path/to/keystore.jks'      # Set the path to the keystore to be used, if it is required by the Logstash input protocol.
     #   keystorePass: 'P4s5w0rd'                   # Set the password, or 'keystore' credential alias to the password, of the keystore.
     #   keyAlias: ''                               # Set the alias name of the private key, if mutual authentication is required by the Logstash input protocol.
     #   KeyPassword: ''                                # Set the password, or 'keystorekey' credential alias to the password, for accessing the private mutual authentication key.
     #   truststoreFile: '/path/tp/truststore.jks'  # Set the path to the truststore to be used, if it is required by the Logstash input protocol.
     #   truststorePass: 'P4s5w0rd'                 # Set the password, or 'truststore' credential alias to the password, for accessing the truststore.

   For example, to connect to an unsecured Logstash input plug-in by using the beats input protocol, on localhost port 5044, and to upload the log data every 30 seconds, set the following properties:

   
   ELKConnections:
     elkbeats:
       elkProtocol: 'beats'
       hostname: 'localhost'
       port: 5044
       uploadIntervalMilliSecs: 30000 
   

4. Optional: If you are connecting to a secured Logstash input plug-in, configure the security credentials that are required to access it. For example, if you are using HTTP to connect to a Logstash input plug-in that requires a username and password, configure the integration node or server to use an elk credential for basic authentication (basicAuth), by completing the following steps:
   1. Update the node.conf.yaml or server.conf.yaml file to specify a credential to be used for basic authentication (basicAuth), by specifying the credential name in the elkCredential property; for example:

       elkCredential: 'elk_ID'   

   2. Use the mqsicredentials command to create the security credential that you specified in the .conf.yaml file (elk_ID), setting the username and password, and a credential type of elk:

      mqsicredentials --work-dir elk_work_dir --create --credential-name elk_ID --credential-type elk --username user1 --password passw0rd1

5. Enable App Connect Enterprise BIP messages to be sent to the configured ELK Connection, by setting properties in the Log section of the .conf.yaml file:

   
   Log:
     #consoleLog: true          # Control writing BIP messages to standard out. Set to true or false, default is true.
     #outputFormat: 'text'      # Control the format of BIP messages written to standard out and file. Set to ibmjson or text, default is text.
     #eventLog: '[iib.system-work-dir]/log/[iib.system-node-label].[iib.system-server-label].events.txt'     # Control writing BIP messages to file. Set to '' to disable, default is as shown.
     #eventLogFileSize: 10      # The maximum size in MB of an event log file before it is rotated into a new file
     #eventLogFileCount: 10     # The maximum number of event log files that should be rotated between.
     #elkLog: false             # Control the publication of BIP messages to an ELK (Elasticsearch, Logstash, Kibana) stack. Set to true or false, default is false.
     #elkConnections: ''        # Name of the ELK connection to use, for example 'elkConnection1'
                                # Each named ELK Connection must be defined in the ELKConnections section below.
   

   For example, to enable BIP messages to be sent to the ELK Connection that was configured in the previous example (elkbeats), set the following properties:

   
   Log:
     elkLog: true
     elkConnections: 'elkbeats'

6. Restart the integration server for the changes to take effect.
   The integration server's log events (BIP messages) will now be sent to the configured Logstash input plug-in.

   When the integration server starts delivering BIP messages to the configured Logstash input, a confirmation message is written to the log; for example:

   BIP6503I: ( CHECK.IS2 ) The integration server successfully sent data to ELK connection ''ELKbeats'' 
   using elkProtocol ''beats'', hostname ''localhost'' and port ''5444''.

   If the integration server is unable to deliver event data to the configured Logstash input, a BIP message is logged to the integration server's local event log; for example:

   BIP3888E: ( CHECK.IS2 ) The ELK connector ''ELKbeats'' failed to send data to ''localhost:5444''. 
   Error details: ''SocketException BIP3150E: ImbBasicSocket::connectTimeout "An error occurred whilst performing a socket operation: 
   getsockopt" [::connect::select(), 10061, No connection could be made because the target machine actively refused it.