Data formats by stage

Data format support

This appendix lists the data formats supported by origin, processor, and destination stages.

Origins

In the following table, a checkmark indicates the data formats supported by each origin.

Origin	Avro	Binary	Datagram	Delimited	Excel	JSON	Log	Parquet	Protobuf	SDC Record	Text	Whole File	XML
Amazon S3	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Amazon SQS Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
Aurora PostgreSQL CDC Client	* * * Not Applicable * * *
Azure Blob Storage	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure Data Lake Storage Gen2	✓			✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure Data Lake Storage Gen2 (Legacy)	✓			✓	✓	✓	✓		✓	✓	✓	✓	✓
Azure IoT/Event Hub Consumer		✓				✓				✓	✓
CoAP Server		✓	✓	✓		✓	✓		✓	✓	✓		✓
CONNX	* * * Not Applicable * * *
CONNX CDC	* * * Not Applicable * * *
Couchbase	* * * Not Applicable * * *
Cron Scheduler	* * * Not Applicable * * *
Directory	✓	✓		✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Elasticsearch	* * * Not Applicable * * *
File Tail						✓	✓				✓
Google BigQuery	* * * Not Applicable * * *
Google Cloud Storage	✓	✓		✓	✓	✓		✓	✓	✓	✓	✓
Google Pub/Sub Subscriber	✓	✓		✓		✓			✓	✓	✓		✓
Groovy Scripting	* * * Not Applicable * * *
Hadoop FS Standalone	✓			✓	✓	✓	✓		✓	✓	✓	✓	✓
HTTP Client		✓		✓		✓	✓			✓	✓		✓
HTTP Server	✓	✓	✓	✓		✓			✓	✓			✓
IBM Db2	* * * Not Applicable * * *
JavaScript Scripting	* * * Not Applicable * * *
JDBC Multitable Consumer	* * * Not Applicable * * *
JDBC Query Consumer	* * * Not Applicable * * *
Jira	* * * Not Applicable * * *
JMS Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
Jython Scripting	* * * Not Applicable * * *
Kafka Multitopic Consumer	✓	✓	✓	✓		✓	✓		✓	✓	✓		✓
Kinesis Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
MapR DB CDC	* * * Not Applicable * * *
MapR DB JSON	* * * Not Applicable * * *
MapR FS Standalone	✓			✓	✓	✓	✓		✓	✓	✓	✓	✓
MapR Multitopic Streams Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
MapR Streams Consumer	✓	✓		✓		✓	✓		✓	✓	✓
MongoDB	* * * Not Applicable * * *
MongoDB Atlas	* * * Not Applicable * * *
MongoDB Atlas CDC	* * * Not Applicable * * *
MongoDB Oplog	* * * Not Applicable * * *
MQTT Subscriber		✓	✓	✓		✓	✓		✓	✓	✓		✓
MySQL Binary Log	* * * Not Applicable * * *
OPC UA Client	* * * Not Applicable * * *
Oracle Bulkload	* * * Not Applicable * * *
Oracle CDC	* * * Not Applicable * * *
Oracle CDC Client	* * * Not Applicable * * *
Oracle Multitable Consumer	* * * Not Applicable * * *
PostgreSQL CDC Client	* * * Not Applicable * * *
Pulsar Consumer	✓	✓	✓	✓		✓	✓		✓	✓	✓		✓
Pulsar Consumer (Legacy)	✓	✓	✓	✓		✓	✓		✓	✓	✓		✓
RabbitMQ Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
Redis Consumer	✓	✓		✓		✓	✓		✓	✓	✓		✓
REST Service	✓	✓	✓	✓		✓			✓	✓			✓
Salesforce	* * * Not Applicable * * *
Salesforce Bulk API 2.0	* * * Not Applicable * * *
SAP HANA Query Consumer	* * * Not Applicable * * *
SFTP/FTP/FTPS Client	✓			✓	✓	✓	✓		✓	✓	✓	✓	✓
Snowflake Bulk	* * * Not Applicable * * *
SQL Server CDC Client	* * * Not Applicable * * *
SQL Server Change Tracking	* * * Not Applicable * * *
Start Jobs	* * * Not Applicable * * *
TCP Server	✓	✓		✓		✓	✓		✓	✓	✓		✓
UDP Multithreaded Source	* * * Not Applicable * * *
UDP Source	* * * Not Applicable * * *
Web Client	✓	✓	✓	✓		✓	✓		✓		✓		✓
WebSocket Client		✓	✓	✓		✓	✓		✓	✓	✓		✓
WebSocket Server		✓	✓	✓		✓	✓		✓	✓	✓		✓

Processors

In the following table, a checkmark indicates the data formats each processor reads.

Processor	Avro	Binary	Datagram	Delimited	JSON	Log	Netflow	Protobuf	SDC Record	Syslog	Text	XML
Data Parser	✓			✓	✓	✓	✓	✓		✓		✓
HTTP Client	✓	✓	✓	✓	✓	✓		✓	✓		✓	✓
JSON Parser					✓
Kaitai Struct Parser		✓
Log Parser						✓
Web Client	✓	✓	✓	✓	✓	✓		✓			✓	✓
XML Parser												✓

In the following table, a checkmark indicates the data formats each processor writes.

Processor	Avro	Binary	Delimited	JSON	Log	Protobuf	SDC Record	Text	XML
Data Generator	✓	✓	✓	✓		✓	✓	✓	✓
JSON Generator				✓

Destinations

In the following table, a checkmark indicates the data formats supported by each destination.

Destination	Avro	Binary	Delimited	JSON	Protobuf	Parquet	SDC Record	Text	Whole File	XML
Aerospike Client	* * * Not Applicable * * *
Amazon S3	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure Blob Storage	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure Data Lake Storage Gen2	✓	✓	✓	✓	✓	✓	✓	✓	✓
Azure Event Hub Producer		✓		✓			✓	✓		✓
Azure IoT Hub Producer		✓		✓			✓	✓
Azure Synapse SQL	* * * Not Applicable * * *
Cassandra	* * * Not Applicable * * *
CoAP Client		✓		✓			✓	✓
Couchbase	✓	✓	✓	✓	✓		✓	✓
Databricks	* * * Not Applicable * * *
Elasticsearch	* * * Not Applicable * * *
Google BigQuery	* * * Not Applicable * * *
Google Bigtable	* * * Not Applicable * * *
Google Cloud Storage	✓		✓	✓	✓	✓	✓	✓	✓
Google Pub/Sub Publisher	✓	✓	✓	✓	✓		✓	✓		✓
Hadoop FS	✓	✓	✓	✓	✓	✓	✓	✓	✓
HBase	* * * Not Applicable * * *
Hive Metastore	✓
HTTP Client	✓	✓	✓	✓	✓		✓	✓
IBM Cloud Object Storage	* * * Not Applicable * * *
IBM Db2	* * * Not Applicable * * *
IBM watsonx.data	* * * Not Applicable * * *
InfluxDB 2.x	* * * Not Applicable * * *
JDBC Producer	* * * Not Applicable * * *
Jira	* * * Not Applicable * * *
JMS Producer	✓	✓	✓	✓	✓		✓	✓		✓
Kafka Producer	✓	✓	✓	✓	✓		✓	✓		✓
Kinesis Firehose			✓	✓
Kinesis Producer	✓	✓	✓	✓	✓		✓	✓
Kudu	* * * Not Applicable * * *
Local FS	✓		✓	✓	✓	✓	✓	✓	✓
MapR DB	* * * Not Applicable * * *
MapR DB JSON	* * * Not Applicable * * *
MapR FS	✓		✓	✓	✓		✓	✓	✓
MapR Streams Producer	✓	✓	✓	✓	✓		✓	✓
MongoDB	* * * Not Applicable * * *
MongoDB Atlas	* * * Not Applicable * * *
MQTT Publisher		✓		✓			✓	✓
Named Pipe			✓	✓			✓	✓
Oracle	* * * Not Applicable * * *
Pulsar Producer	✓	✓	✓	✓	✓		✓	✓		✓
RabbitMQ Producer	✓	✓	✓	✓	✓		✓	✓
Redis	✓	✓	✓	✓	✓		✓	✓
Salesforce	* * * Not Applicable * * *
Salesforce Bulk API 2.0	* * * Not Applicable * * *
Send Response to Origin							✓
SFTP/FTP/FTPS Client									✓
SingleStore	* * * Not Applicable * * *
Snowflake	* * * Not Applicable * * *
Snowflake File Uploader									✓
Solr	* * * Not Applicable * * *
Splunk	* * * Not Applicable * * *
SQL Server 2019 BDC Bulk Loader	* * * Not Applicable * * *
Syslog	✓	✓	✓	✓	✓		✓	✓		✓
Tableau CRM	* * * Not Applicable * * *
Teradata	* * * Not Applicable * * *
To Error	* * * Not Applicable * * *
Trash	* * * Not Applicable * * *
Web Client	✓	✓	✓	✓	✓	✓	✓	✓	✓	✓
WebSocket Client		✓		✓			✓	✓