Syslog
The Syslog destination writes syslog messages to a Syslog server.
For example, you might have network devices that write log messages to flat files. You need to send these flat files to a centralized Syslog server, but the messages first must be converted to syslog messages. You design a pipeline that reads the flat files, performs some processing on the data, and then uses a Syslog destination that formats the data into syslog messages and writes the messages to the Syslog server.
When you configure the Syslog destination, you define the host name and port number of the Syslog server. You also define whether to use the UDP or TCP protocol to send messages to the server. If using the TCP protocol, you can optionally enable SSL or TLS for the destination.
You construct the syslog message content by specifying expressions that evaluate to the message values - such as the timestamp, facility code, severity level, message ID, and the log message itself.
You'll typically want to write the log message using the text data format. However, if needed, you can configure the Syslog destination to use a different data format for the log message.
Protocol
Using SSL/TLS
You can configure the Syslog destination to use SSL, TLS, and mutual TLS encryption. By default, the destination does not use SSL or TLS.
- SSL
- To use SSL encryption, the Syslog destination requires configuring Java configuration options in the deployment in addition to configuring the destination to use SSL. For more information, see Enabling SSL.
- TLS
- You can configure the destination to use TLS by selecting the Enable TLS property. SSL must be enabled before you can enable TLS.
- Mutual TLS
- You can configure the destination to use mutual TLS encryption by selecting the Enable Mutual TLS property. SSL and TLS must be enabled before you can enable mutual TLS.
Enabling SSL
If you configure the Syslog destination to use the TCP protocol, you can enable the destination to use SSL to connect to the Syslog server.
Message Content
A syslog message includes fields such as a timestamp, facility code, severity level, message ID, and the log message itself. You construct the syslog message content by specifying the values for message fields on the Message tab. The content of the log message depends on how you configure the Data Format tab.
Use the Message tab to construct the syslog message properties by entering values or expressions for the message properties. In most cases, you'll want to define expressions to call fields in the record that contain the message values.
For example, the following image shows the timestamp and host name message properties defined with expressions and the severity level and syslog facility set to the default values of 6 and 1:
The content of the log message itself depends on the data format selected on the Data Format tab. For the Text data format, the destination writes data from a single text field in the record to the log message field. For the Binary data format, the destination writes data from a single field in the record to the log message field. For the remaining data formats, the destination writes data from all fields in the record to the log message field.
Data Formats
The Syslog destination writes data to the log message based on the data format that you select.
The destination can use the following data formats:
- Avro
- The stage writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
- XML
- The destination creates a valid XML document for each record. The
destination requires the record to have a single root field that
contains the rest of the record data. For details and
suggestions for how to accomplish this, see Record Structure Requirement.
The destination can include indentation to produce human-readable documents. It can also validate that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling configured for the destination.
Configuring a Syslog Destination
Configure a Syslog destination to write syslog messages to a Syslog server.