Installing Fluentd on the Linux Operating Systems

You need to install Fluentd to use the Log search console.

1. Check and increase the maximum number of file descriptors and then restart the Linux® system by using the following command.

   $: sudo reboot

2. Optimize the network kernel parameters.
3. Install Fluentd by using the following command.

   Red Hat®, CentOS, or Rocky

   $: curl -fsSL https://toolbelt.treasuredata.com/sh/install-redhat-fluent-package5-lts.sh | sh

   Ubuntu

   $: curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-fluent-package5-lts.sh | sh

4. Update the /etc/fluent/fluentd.conf file.

   cat > /etc/fluent/fluentd.conf << EOF
   <source>
     @type tail
     # Add the comma-separated location of your log file
     path <comma-separated log file locations>
     pos_file /var/log/fluent/ipm.log.pos
     read_from_head true
     tag ipm.logs
     <parse>
       @type multiline
       format_firstline /^\d{4}-\d{2}-\d{2}/
       format1 /^(?<logdatetime>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?<service>\S+) (?<level>[^\s]+)(?: )? (?<logger>[^?]+) \[(?<thread>.+)\] \[log_file:(?<logfile>.*?)\] \[(?:cmp:)?(?<cmpId>-?\d+|NA)\](?: ip_address=\[(?<ip_address>[^\]]+)\])?(?: user=\[(?<user>[^\]]*)\])? (?<message>.*)(?=\s*\S{0,}\s\w*)?/
     </parse>
   </source>
   
   <filter ipm.logs>
     @type record_transformer
     enable_ruby true
     <record>
       logdatetime ${record['logdatetime'].split(' ').join('T')}
       message ${record['message'].gsub(/[\s]+/, ' ')}
     </record>
   </filter>
   
   <match ipm.logs>
     @type opensearch
     # Add OpenSearch Server IP address
     hosts <OpenSearch Server IP address>
     logstash_format true
     logstash_dateformat %Y_%m_%d
     logstash_prefix log
     <buffer>
       flush_interval 10s
     </buffer>
     <format>
       @type json
     </format>
   </match>
   EOF

5. Optional: If you want to use HTTPS as a prefix in the OpenSearch server IP address, add the following configuration to the fluentd.config file.

   host <OpenSearch_IP>
      port 9200
      scheme https
      user <OpenSearch_user>
      password <OpenSearch_password>
      ssl_verify false
      ssl_version TLSv1_2

6. Launch Daemon and verify the status for the Fluentd.
   You can check the version by using the following command.

   $: fluentd –version

   You can check the logs by using the following command.

   $: tail -f /var/log/fluent/fluentd.log