SSQYQIJ - Documentation Index

Table of Contents

• Welcome
• Getting Started
  • Meet StreamSets Control Hub
    • Design the Complete Data Architecture
    • Collaboratively Build Pipelines
    • Execute Jobs at Scale
    • Map Jobs into a Topology
    • Measure Dataflow Quality
    • Monitor Dataflow Operations
  • Data Collector Communication
    • Data Collector Requests
  • Transformer Communication
    • Transformer Requests
  • Provisioning Agent Communication
    • Provisioning Agent Requests
  • Control Hub Requirements
    • Browser Access to Data Collector and Transformer
    • Control Hub IP Addresses
  • Control Hub User Interface
    • Dashboards
      • Alerts Dashboard
      • Topologies Dashboard
    • Alerts
    • My Account
• Release Notes
  • May 15, 2026
  • February 13, 2026
  • December 17, 2025
  • November 14, 2025
  • August 8, 2025
  • April 7, 2025
  • January 24, 2025
  • October 4, 2024
  • July 26, 2024
  • May 24, 2024
  • April 5, 2024
  • February 9, 2024
  • December 8, 2023
  • October 6, 2023
  • September 8, 2023
  • July 26, 2023
  • May 19, 2023
  • April 7, 2023
  • January 30, 2023
  • January 27, 2023
  • November 11, 2022
  • August 19, 2022
  • July 22, 2022
  • June 24, 2022
  • June 10, 2022
  • March 18, 2022
  • February 18, 2022
  • January 7, 2022
  • December 10, 2021
  • September 22, 2021
  • August 27, 2021
  • June 30, 2021
  • June 25, 2021
  • May 14, 2021
  • April 16, 2021
  • February 19, 2021
  • February 12, 2021
  • February 5, 2021
  • December 11, 2020
  • November 13, 2020
  • November 6, 2020
  • September 25, 2020
  • July 22, 2020
  • July 17, 2020
  • May 29, 2020
  • May 11, 2020
  • May 8, 2020
  • April 8, 2020
  • March 21, 2020
  • January 10, 2020
  • December 16, 2019
  • November 23, 2019
  • November 8, 2019
  • October 25, 2019
  • October 11, 2019
  • September 27, 2019
  • September 20, 2019
  • September 15, 2019
  • September 4, 2019
  • August 30, 2019
  • June 14, 2019
    • Update Deployments for Provisioned Data Collectors
  • February 27, 2019
  • December 21, 2018
  • November 28, 2018
  • November 19, 2018
  • October 27, 2018
  • October 12, 2018
  • October 4, 2018
  • September 28, 2018
  • August 29, 2018
  • August 4, 2018
  • May 25, 2018
  • May 11, 2018
  • March 30, 2018
  • March 6, 2018
  • January 14, 2018
  • December 15, 2017
  • September 22, 2017
  • August 9, 2017
  • June 17, 2017
  • April 15, 2017
  • March 4, 2017
  • Known Issues
• Tutorials
  • Tutorial: Data Collectors, Pipelines, and Jobs
    • Before You Begin
    • Register a Data Collector
    • Assign Labels to the Data Collector
    • Design a Pipeline
    • Publish the Pipeline
    • Add a Job for the Pipeline
    • Start and Monitor the Job
  • Tutorial: Topologies
    • Design and Publish Related Pipelines
    • Add Jobs for the Related Pipelines
    • Map Jobs in a Topology
    • Measure and Monitor the Topology
• Pipeline Repository
  • Pipeline Repository Management
    • Overview
      • Sample Pipelines View
    • Design in Control Hub
      • Authoring Engine
        • Data Collector Stage Libraries
        • External Libraries
      • Pipeline Designer UI
      • Pipeline Designer Tips
      • Creating a Pipeline
      • Creating a Fragment
      • Running a Test of a Draft Pipeline
      • Publishing a Fragment or Pipeline
    • Pipeline Labels
      • Deleting Pipeline Labels
    • Sample Pipelines
      • Creating User-Defined Sample Pipelines
    • Version History
      • Versions
        • Comparing Versions
        • Deleting Versions
      • Tags
        • Creating Tags
        • Removing Tags
    • Filtering Pipelines and Fragments
    • Viewing Published Pipelines and Fragments
    • Duplicating a Pipeline or Fragment
    • Creating a Job for a Pipeline
    • Viewing Jobs that Include a Pipeline
    • Deleting Versions
      • Deleting a Draft Version
    • Troubleshooting
  • Fragments
    • Fragments Overview
    • Fragment Configuration
      • Creating Pipeline Fragments
        • Using a Blank Canvas
        • Using Pipeline Stages
      • Fragment Input and Output
      • Execution Engine and Execution Mode
      • Data and Data Drift Rules and Alerts
      • Runtime Parameters
        • Prefix for Runtime Parameters
        • Using Runtime Parameters
      • Creating Additional Streams
      • Data Preview
      • Explicit Validation
      • Fragment Publishing Requirements
    • Fragment Types and Default Pipeline Labels
    • Using Fragments in Pipelines
      • Using Fragment Versions
      • Stream Order
    • Fragments Tips and Best Practices
      • Design Best Practices
      • Usage Tips
• Pipeline Design (Data Collector)
  • Pipeline concepts and design
    • What is a Data Collector pipeline?
    • Data in motion
      • Single and multithreaded pipelines
      • Delivery guarantee
      • Data Collector data types
    • Designing the data flow
      • Branching streams
      • Merging streams
    • Dropping unwanted records
      • Required fields
      • Preconditions
    • Error record handling
      • Pipeline error record handling
      • Stage error record handling
      • Example
      • Error records and version
    • Record header attributes
      • Working with header attributes
        • Internal attributes
      • Header attribute-generating stages
      • Record header attributes for record-based writes
        • Generating attributes for record-based writes
      • Viewing attributes in data preview
    • Field attributes
      • Working with field attributes
      • Field attribute-generating stages
      • Viewing field attributes in data preview
    • Processing changed data
      • CRUD operation header attribute
        • Earlier implementations
      • CDC-enabled stages
      • Data changes in SQL Server or Azure SQL Database
      • CRUD-enabled stages
      • Processing the record
      • Use cases
    • Control character removal
    • Development stages
    • Shortcut keys for pipeline design
    • Test origin for preview
      • Configuring a test origin
      • Using a test origin in data preview
    • Understanding pipeline states
      • State transition examples
    • Technology preview features
  • Pipeline configuration
    • Retrying the pipeline
    • Rate limit
    • Advanced options
    • Simple and bulk edit mode
    • Runtime values
      • Runtime parameters
        • Step 1. Define runtime parameters
        • Step 2. Call the runtime parameters
          • Calling from text boxes
          • Calling from checkboxes and drop-down menus
          • Calling from scripting processors
      • Runtime resources
        • Step 1. Define runtime resources
        • Step 2. Call the runtime resource
      • Runtime properties
        • Step 1. Define runtime properties
        • Step 2. Call the runtime property
    • Event generation
      • Pipeline event records
    • Webhooks
      • Request method
      • Payload and parameters
      • Examples
    • Notifications
    • SSL/TLS encryption
      • Keystore and truststore configuration
        • Local keystore and truststore
        • Remote keystore and truststore
          • Using a credential store
      • Transport protocols
      • Cipher suites
    • Security in Amazon stages
      • Assume another role
        • Assume role methods
        • Create the trust policy
        • Configure stages to assume a role
    • Security in Google Cloud stages
      • Default credentials
      • Credentials in a file
      • Credentials in a property
    • Security in Kafka stages
      • Prerequisite tasks
      • SASL authentication credentials
        • Providing PLAIN credentials
        • Providing Kerberos credentials
          • Using a credential store
      • Enabling SSL/TLS encryption
      • Enabling SSL/TLS encryption and authentication
      • Enabling SASL authentication
      • Enabling SASL authentication on SSL/TLS
      • Enabling Confluent Cloud authentication
      • Enabling custom authentication
    • Kafka message keys
      • Storing message keys
      • Message key formats
        • String message keys
        • Avro message keys
      • Passing key values to Kafka
      • Example: Storing and passing Kafka message keys
    • SSL/TLS in CONNX stages
    • Authentication in Salesforce stages
      • Connected app with OAuth prerequisites
    • Considerations for IBM Connectivity Service stages
      • IBM Connectivity Service stage prerequisites
      • Troubleshooting IBM Connectivity Service
    • Implicit and explicit validation
    • Expression configuration
      • Basic syntax
      • Using field names in expressions
        • Field names with special characters
      • Referencing field names and field paths
        • Wildcard use for arrays and maps
      • Field path expressions
        • Supported stages
        • Field path expression syntax
      • Data type coercion
    • Configuring a pipeline
  • Data formats
    • Data formats overview
      • File compression formats
    • Avro data format
      • Reading Avro data
      • Writing Avro data
    • Binary data format
    • Datagram data format
    • Delimited data format
      • Reading delimited data
        • CSV parser
        • Delimited data root field type
      • Writing delimited data
    • Excel data format
    • Log data format
    • NetFlow data processing
      • Caching NetFlow 9 templates
      • NetFlow 5 generated records
      • NetFlow 9 generated records
    • Parquet data format
      • Reading Parquet data
      • Writing Parquet data
    • Protobuf data format prerequisites
    • SDC record data format
    • Text data format with custom delimiters
      • Processing XML data with custom delimiters
    • Whole file data format
      • Basic pipeline
      • Whole file records
        • Processors for file reference information
      • Defining the transfer rate
      • Writing whole files
        • Access permissions
        • Including checksums in events
    • Reading and processing XML data
      • Creating multiple records with an XML element
        • Using XML elements with namespaces
      • Creating multiple records with an XPath expression
        • Using XPath expressions with namespaces
        • Simplified XPath syntax
        • Sample XPath expressions
        • Predicates in XPath expressions
        • Predicate examples
      • Preserving the root element
      • Including field XPaths and namespaces
      • XML attributes and namespace declarations
      • Parsed XML
    • Writing XML data
      • Record structure requirement
  • Origins
    • Origins
      • Comparing Azure storage origins
      • Comparing HTTP origins
      • Comparing MapR origins
      • Comparing UDP Source origins
      • Comparing WebSocket origins
      • Batch size and wait time
      • Maximum record size
      • Previewing raw source data
    • Amazon S3
      • Authentication method
      • Common prefix, prefix pattern, and wildcards
      • Multithreaded processing
      • Record header attributes
        • Object metadata in record header attributes
      • Read order
      • Buffer limit and error handling
      • Server side encryption
      • Event generation
        • Event records
      • Data formats
      • Configuring an Amazon S3 origin
    • Amazon SQS Consumer
      • Authentication method
      • Queue name prefix
      • Multithreaded processing
      • Including SQS message attributes
        • Including sender attributes
      • Data formats
      • Configuring an Amazon SQS Consumer origin
    • Aurora PostgreSQL CDC Client
      • Aurora PostgreSQL prerequisites
        • Enable logical replication
        • Assign the required role
      • JDBC driver
      • Schema, table name, and exclusion patterns
      • Initial change
      • Memory usage
      • SSL/TLS encryption
      • Record contents and generated records
        • Record header attributes
        • Sample records
      • Configuring an Aurora PostgreSQL CDC Client origin
    • Azure Blob Storage
      • Prerequisites
      • Common path, path pattern, and wildcards
      • Read order
      • Buffer limit and error handling
      • Multithreaded processing
      • Record header attributes
        • Object metadata in record header attributes
      • Event generation
        • Event records
      • Data formats
      • Configuring an Azure Blob Storage origin
    • Azure Data Lake Storage Gen2
      • Prerequisites
        • Retrieve authentication information
      • Common path, path pattern, and wildcards
      • Read order
      • Buffer limit and error handling
      • Multithreaded processing
      • Record header attributes
        • Object metadata in record header attributes
      • Event generation
        • Event records
      • Data formats
      • Configuring an Azure Data Lake Storage Gen2 origin
    • Azure Data Lake Storage Gen2 (Legacy)
      • Prerequisites
        • Retrieve authentication information
      • File directory
      • File name pattern and mode
      • Read order
      • Multithreaded processing
      • Reading from subdirectories
        • Post-processing subdirectories
      • First file for processing
      • Record header attributes
      • Event generation
        • Event records
      • Buffer limit and error handling
      • Data formats
      • Configuring an Azure Data Lake Storage Gen2 (Legacy) origin
    • Azure IoT/Event Hub Consumer
      • Storage account and container prerequisite
      • Resetting the origin in Event Hub
      • Multithreaded processing
      • Data formats
      • Configuring an Azure IoT/Event Hub Consumer origin
    • CoAP Server
      • Prerequisites
      • Multithreaded processing
      • Network configuration properties
      • Data formats
      • Configuring a CoAP Server origin
    • Cron Scheduler
      • Cron Expression
      • Generated record
      • Configuring a Cron Scheduler origin
    • Couchbase
      • Prerequisites
      • Offset
      • Event generation
        • Event record
      • Multithreaded processing
      • Configuring a Couchbase origin
    • Directory
      • File name pattern and mode
      • Read order
      • Multithreaded processing
      • Reading from subdirectories
        • Post-processing subdirectories
      • First file for processing
      • Late directory
      • Record header attributes
      • Event generation
        • Event records
      • Buffer limit and error handling
      • Data formats
      • Configuring a Directory origin
    • Elasticsearch
      • Security
      • Batch and incremental mode
      • Query
        • Incremental mode query
      • Scroll timeout
      • Multithreaded processing
      • Configuring an Elasticsearch origin
    • File Tail
      • File processing and archived file names
      • Multiple paths and file sets
      • First file for processing
      • Late directories
      • Files matching a pattern - pattern constant
      • Record header attributes
        • Defining and using a tag
      • Multiple line processing
      • File Tail output
      • Event generation
        • Event records
      • Data formats
      • Configuring a File Tail origin
    • Google BigQuery
      • Credentials
      • BigQuery data types
        • Datetime conversion
      • Event generation
        • Event record
      • Configuring a Google BigQuery origin
    • Google Cloud Storage
      • Credentials
      • Common prefix, prefix pattern, and wildcards
      • Record header attributes
      • Event generation
        • Event records
      • Data formats
      • Configuring a Google Cloud Storage origin
    • Google Pub/Sub Subscriber
      • Credentials
      • Multithreaded processing
      • Record header attributes
      • Data formats
      • Configuring a Google Pub/Sub Subscriber origin
    • Groovy Scripting
      • Scripting objects
      • Multithreaded processing
      • Accessing record details
      • Type handling
      • Event generation
        • Event record
      • Record header attributes
      • Calling external Java code
      • Granting permissions on Groovy scripts
      • Configuring a Groovy Scripting origin
    • Hadoop FS Standalone
      • File directory
      • File name pattern and mode
      • Read order
      • Multithreaded processing
      • Reading from subdirectories
        • Post-processing subdirectories
      • First file for processing
      • Reading from Azure Blob Storage
      • Reading from Azure Blob Storage with Azure HDInsight
      • Record header attributes
      • Event generation
        • Event records
      • Buffer limit and error handling
      • Kerberos authentication
      • HDFS properties and configuration files
      • Impersonation user
      • Data formats
      • Configuring a Hadoop FS Standalone origin
    • HTTP Client
      • Processing mode
      • HTTP method
      • Headers
      • Per-status actions
      • Pagination
        • Page or offset number
        • Result field path
        • Keep all fields
        • Pagination examples
          • Example for link in HTTP header
          • Example for link in response field
          • Example for page number
          • Example for offset number
      • OAuth 2 authorization
        • Example for Twitter
        • Example for Microsoft Azure AD
        • Example for Google
      • Logging request and response data
      • Generated records
      • Data formats
      • Configuring an HTTP Client origin
    • HTTP Server
      • Prerequisites
        • Send data to the listening port
        • Include the application ID in requests
      • Multithreaded processing
      • Data formats
      • Record header attributes
      • Configuring an HTTP Server origin
    • IBM Db2
      • Prerequisites
      • Multithreaded processing
      • Record header attributes
      • Field attributes
      • IBM Db2 data types
        • Key column data types
      • Troubleshooting IBM Connectivity Service
      • Configuring an IBM Db2 origin
    • JavaScript Scripting
      • Scripting objects
      • Multithreaded processing
      • Accessing record details
      • Type handling
      • Event generation
        • Event record
      • Record header attributes
      • Calling external Java code
      • Configuring a JavaScript Scripting origin
    • JDBC Multitable Consumer
      • Database vendors and drivers
        • MySQL data types
        • Oracle data types
        • PostgreSQL data types
        • SQL Server data types
        • Unsupported data types
      • Installing the JDBC driver
        • Working with a MySQL JDBC driver
      • Table configuration
        • Schema, table name, and exclusion patterns
        • Offset column and value
        • Reading from views
      • Multithreaded processing modes
        • Multithreaded table processing
        • Multithreaded partition processing
          • Partition processing requirements
          • Multiple offset value handling
          • Best effort: Processing non-compliant tables
      • Non-incremental processing
      • Batch strategy
        • Process all available rows
        • Switch tables
      • Initial table order strategy
      • Processing queue
        • Multithreaded table processing only
        • Multithreaded partition processing only
        • Both multithreaded partition and table processing
      • JDBC attributes
        • JDBC header attributes
        • JDBC field attributes
      • Event generation
        • Event record
      • Configuring a JDBC Multitable Consumer origin
    • JDBC Query Consumer
      • Database vendors and drivers
        • MySQL data types
        • Oracle data types
        • PostgreSQL data types
        • SQL Server data types
        • Unsupported data types
      • Installing the JDBC driver
      • Offset column and offset value
      • Full and incremental mode
      • Recovery
      • SQL query
        • SQL query for incremental mode
        • SQL query for full mode
        • Stored procedure in full mode
      • JDBC attributes
        • JDBC header attributes
          • Header attributes with the drift synchronization solution
        • JDBC field attributes
      • Event generation
        • Event record
      • Configuring a JDBC Query Consumer origin
    • Jira
      • Event generation
        • Event records
      • OAuth 2 authentication
      • Configuring a Jira origin
    • JMS Consumer
      • Installing JMS drivers
      • Additional JMS properties
      • Working with TIBCO EMS
        • Configuring for TIBCO and SSL
      • Data formats
      • Configuring a JMS Consumer origin
    • Jython Scripting
      • Scripting objects
        • Thread safety in Jython scripts
      • Multithreaded processing
      • Accessing record details
      • Type handling
      • Event generation
        • Event record
      • Record header attributes
      • Calling external Java code
      • Configuring a Jython Scripting origin
    • Kafka Multitopic Consumer
      • Offset management
      • Multithreaded processing
      • Additional Kafka properties
      • Record header attributes
      • Kafka security
      • Data formats
      • Configuring a Kafka Multitopic Consumer origin
    • Kinesis Consumer
      • Multithreaded processing
      • Authentication method
      • Additional Kinesis properties
      • Read interval
      • Lease table tags
      • Resetting the Kinesis Consumer origin
      • Data formats
      • Reading from DynamoDB or CloudWatch
      • Configuring a Kinesis Consumer origin
    • MapR DB CDC
      • Multithreaded processing
      • Handling the _id field
      • CRUD operation and CDC header attributes
      • Additional properties
      • Configuring a MapR DB CDC origin
    • MapR DB JSON
      • Handling the _id field
      • Configuring a MapR DB JSON origin
    • MapR FS Standalone
      • File name pattern and mode
      • Read order
      • Multithreaded processing
      • Reading from subdirectories
        • Post-processing subdirectories
      • First file for processing
      • Record header attributes
      • Event generation
        • Event records
      • Buffer limit and error handling
      • Kerberos authentication
      • HDFS properties and configuration files
      • Impersonation user
      • Data formats
      • Configuring a MapR FS Standalone origin
    • MapR Multitopic Streams Consumer
      • Initial and subsequent offsets
      • Processing all unread data
      • Multithreaded processing
      • Additional properties
      • Record header attributes
      • Data formats
      • Configuring a MapR Multitopic Streams Consumer origin
    • MapR Streams Consumer
      • Processing all unread data
      • Data formats
      • Additional properties
      • Record header attributes
      • Configuring a MapR Streams Consumer origin
    • MongoDB Atlas
      • Credentials
      • Offset field and initial offset
      • Specifying field paths
      • Read preference
      • Custom filter
      • Event generation
        • Event records
      • Enabling SSL/TLS
      • MongoDB data types
      • Reading BSON types
      • Configuring a MongoDB Atlas origin
    • MongoDB Atlas CDC
      • Credentials
      • Read preferences
      • Generated records
        • CRUD operation and CDC header attributes
      • Enabling SSL/TLS
      • MongoDB data types
      • Reading BSON types
      • Configuring a MongoDB Atlas CDC origin
    • MQTT Subscriber
      • Topics
      • Record header attributes
      • Data formats
      • Configuring an MQTT Subscriber origin
    • MySQL Binary Log
      • Prerequisites
        • Configure MySQL Server for row-based logging
        • Enable the required authentication
        • Set the row metadata property
        • Install the JDBC driver
      • Initial offset
      • Generated records
      • Processing generated records
      • Tables to include or ignore
      • MySQL data types
      • Configuring a MySQL Binary Log origin
    • OPC UA Client
      • Processing mode
      • Providing NodeIds
      • Security
      • Configuring an OPC UA Client origin
    • Oracle Bulkload
      • Prerequisite
      • Static and non-static tables
      • Batch processing
      • Schema and table names
      • Multithreaded processing
      • Field attributes
      • Event generation
        • Event record
      • Configuring an Oracle Bulkload origin
    • Oracle CDC
      • Comparing Oracle CDC origins
        • Updating Oracle CDC Client pipelines
      • Prerequisites
        • Task 1. Configure the database archiving mode
        • Task 2. Enable supplemental logging
        • Task 3. Create a user account
        • Task 4. Install required libraries on Data Collector
      • Primary and standby databases
      • Table filters
      • Start mode
      • Record formats
        • Custom record options
      • Parse SQL statements, supported operations, and record attributes
        • CRUD operation header attributes
        • CDC header attributes
        • Other header attributes
        • Default field attributes
        • Annotation field attributes
      • Multithreaded parsing
      • Query fetch size
      • Uncommitted transaction handling
        • Evicted transaction handling
      • Event generation
        • Event records
      • Oracle data types
        • Unsupported data types
      • Data preview with Oracle CDC
      • Working with the drift synchronization solution
      • Configuring an Oracle CDC origin
    • Oracle CDC Client
      • LogMiner dictionary source
      • Oracle CDC Client prerequisites
        • Task 1. Enable database archiving mode
        • Task 2. Enable supplemental loggingIf you change this title, update the text reference in Task 1.
        • Task 3. Create a user accountIf you change this title, update the text reference in Task 2.
        • Task 4. Extract a LogMiner dictionary (redo logs)
        • Task 5. Install the driver
      • Available parsers
      • Table configuration
        • Schema, table name, and exclusion patterns
      • Initial change
      • Choosing buffers
        • Local buffer resource requirements
        • Uncommitted transaction handling
      • JDBC fetch options
      • Include nulls
      • Unsupported data types
        • Conditional data type support
      • Processing Blob and Clob columns
      • Parse SQL, supported operations, and generated records
        • CRUD operation header attributes
        • CDC header attributes
        • Other header attributes
        • Field attributes
      • Database time zone
        • Change in daylight saving time
      • Event generation
        • Event records
      • Multithreaded parsing
      • Working with the SQL Parser processor
      • Working with the drift synchronization solution
      • Data preview with Oracle CDC Client
      • Configuring an Oracle CDC Client origin
    • Oracle Multitable Consumer
      • Prerequisite
      • Oracle data types
        • Unsupported data types
      • Table configuration
        • Schema, table name, and exclusion patterns
        • Offset column and value
      • Multithreaded processing modes
        • Multithreaded table processing
        • Multithreaded partition processing
          • Partition processing requirements
          • Multiple offset value handling
          • Best effort: Processing non-compliant tables
      • Non-incremental processing
      • Batch strategy
        • Process all available rows
        • Switch tables
      • Initial table order strategy
      • Processing queue
        • Multithreaded table processing only
        • Multithreaded partition processing only
        • Both multithreaded partition and table processing
      • Generated records
        • CRUD operation header attributes
        • JDBC header attributes
        • JDBC field attributes
      • Event generation
        • Event records
      • Configuring an Oracle Multitable Consumer origin
    • Oracle XStream
      • Oracle prerequisites
        • Requirements
        • Command tools and syntax
        • Task 1. Set database parameters
        • Task 2. Create the CDC user
        • Task 3. Create the outbound server (existing components only)
          • Include all tables
          • Include specified tables
        • Task 4. Start the outbound server (existing components only)
        • Task 5. Prepare the database and tables
        • Task 6. Run verification commands
      • Data Collector prerequisite
      • Operation mode
      • Start mode
      • CRUD operation header attributes
      • CDC header attributes
      • Field attributes
      • Event generation
        • Event records
      • Configuring an Oracle XStream origin
    • PostgreSQL CDC Client
      • PostgreSQL prerequisites
        • Install the logical decoder
        • Assign the required role
      • JDBC driver
      • Schema, table name, and exclusion patterns
      • Initial change
      • Memory usage
      • SSL/TLS encryption
      • Record contents and generated record
        • Record header attributes
        • Sample records
      • Configuring a PostgreSQL CDC Client origin
    • Pulsar Consumer
      • Topics selector
      • Multithreaded processing
      • Offset management
      • Record header attributes
      • Schema properties
      • Security
        • Enabling TLS transport encryption
        • Enabling TLS authentication
        • Enabling JWT authentication
        • Enabling OAuth authentication
      • Data formats
      • Configuring a Pulsar Consumer origin
    • Pulsar Consumer (Legacy)
      • Topics selector
        • Parallel reads of a topic
      • Offset management
      • Record header attributes
      • Schema properties
      • Security
        • Enabling TLS transport encryption
        • Enabling TLS authentication
        • Enabling JWT authentication
        • Enabling OAuth authentication
      • Data formats
      • Configuring a Pulsar Consumer (Legacy) origin
    • RabbitMQ Consumer
      • Queue
      • Record header attributes
      • Data formats
      • Configuring a RabbitMQ Consumer origin
    • Redis Consumer
      • Channels and patterns
      • Data formats
      • Configuring a Redis Consumer origin
    • REST Service
      • HTTP listening port
      • API gateway
        • Gateway authentication
        • REST API URLs
        • Required request header
      • Security
      • Using application IDs
      • REST API client requirement
      • Multithreaded processing
      • Generated response
        • Sample responses
      • Record header attributes
      • Data formats
      • Configuring a REST Service origin
    • Salesforce
      • Querying data
        • Using the SOAP and Bulk API
          • Bulk API example
          • Aggregate functions in SOQL queries
        • Using the Bulk API with PK Chunking
          • PK chunking example
        • Repeat query
        • Processing deleted records
      • Subscribing to notifications
        • Processing PushTopic events
          • PushTopic event record format
        • Processing platform events
        • Processing change events
      • Reading custom objects or fields
      • Salesforce attributes
        • Salesforce header attributes
        • CRUD operation header attribute
        • Salesforce field attributes
      • Event generation
        • Event record
      • Changing the API version
      • Configuring a Salesforce origin
    • Salesforce Bulk API 2.0
      • Querying data
        • Bulk API 2.0 queries
      • Full and incremental mode
      • Multithreaded processing
      • Processing deleted records
      • Reading custom objects or fields
      • Salesforce attributes
        • Salesforce header attribute
        • Salesforce field attributes
      • Event generation
        • Event record
      • Changing the API version
      • Configuring a Salesforce Bulk API 2.0 origin
    • SAP HANA Query Consumer
      • Installing the JDBC driver
      • Offset column and offset value
      • Full and incremental mode
      • Recovery
      • SQL query
        • SQL query for incremental mode
        • SQL query for full mode
      • JDBC attributes
        • JDBC header attributes
        • JDBC field attributes
      • SAP HANA header attributes
      • Event generation
        • Event record
      • Configuring an SAP HANA Query Consumer origin
    • SFTP/FTP/FTPS Client
      • File name pattern and mode
      • Read order
      • First file for processing
      • Credentials
      • Record header attributes
      • Event generation
        • Event records
      • Post processing
      • Data formats
      • Configuring an SFTP/FTP/FTPS Client origin
    • Snowflake Bulk
      • Prerequisites
        • Create a Snowflake stage
          • AWS credentials
          • Google Cloud Storage credentials
        • Assign privileges
      • Batch processing
      • Define a role
      • Multithreaded processing
      • Table configurations
      • Error handling
      • Record header attributes
      • Snowflake data types
      • Configuring a Snowflake Bulk origin
    • SQL Server CDC Client
      • JDBC driver
      • Supported operations
      • Time windows
      • Table configuration
      • Initial table order strategy
      • Allow late table processing
      • Checking for schema changes
      • Generated record
        • Record header attributes
        • CRUD operation header attributes
        • Field attributes
      • Event generation
        • Event record
      • Configuring a SQL Server CDC Client origin
    • SQL Server Change Tracking
      • JDBC driver
      • Prerequisites
        • Enable change tracking
        • Assign permissions
      • Table configuration
      • Initial table order strategy
      • Generated record
        • Record header attributes
        • CRUD operation header attributes
        • Field attributes
      • Event generation
        • Event record
      • Configuring a SQL Server Change Tracking origin
    • Start Jobs
      • Job execution and data flow
      • Generated record
      • Runtime parameters
      • Suffix for job instance names
      • Configuring a Start Jobs origin
    • TCP Server
      • Multithreaded processing
      • Closing connections for invalid data
      • Sending acknowledgements
        • Using expressions in messages
      • TCP modes
      • Data formats
      • Configuring a TCP Server origin
    • UDP Multithreaded Source
      • Processing raw data
      • Receiver and worker threads
      • Packet queue
      • Multithreaded pipelines
      • Metrics for performance tuning
      • Configuring a UDP Multithreaded Source
    • UDP Source
      • Processing raw data
      • Receiver threads
      • Configuring a UDP Source
    • Web Client
      • Ingestion mode
      • HTTP method
        • Expression method
      • Headers
      • Grouping style
      • Event generation
        • Event records
      • Per-status actions
      • Per-timeout actions
      • Pagination
        • Page or offset number
      • OAuth 2 authentication
      • Generated records
      • Data formats
      • Configuring a Web Client origin
    • WebSocket Client
      • Read REST response data from Data Collector
      • Generated microservice responses
        • Sample responses
      • Data formats
      • Configuring a WebSocket Client origin
    • WebSocket Server
      • Prerequisites
        • Send data to the listening port
        • Include the application ID in requests
      • Multithreaded processing
      • Generated microservice responses
        • Sample responses
      • Data formats
      • Configuring a WebSocket Server origin
  • Processors
    • Processors
    • Base64 Field Decoder
      • Configuring a Base64 Field Decoder processor
    • Base64 Field Encoder
      • Configuring a Base64 Field Encoder processor
    • Control Hub API
      • Finding the URL to a Control Hub REST API
      • HTTP method
      • Parallel requests
      • Logging request and response data
      • Generated record
      • Configuring a Control Hub API processor
    • Couchbase Lookup
      • Record header attributes
      • Configuring a Couchbase Lookup processor
    • Data Generator
      • Target field
      • Data formats
      • Configuring a Data Generator processor
    • Data Parser
      • Data formats
      • Configuring a Data Parser processor
    • Delay
      • Configuring a Delay processor
    • Encrypt and Decrypt Fields
      • Supported data types
      • Key provider
        • AWS credentials
      • Cipher suite
      • Encryption contexts
      • Data key caching
      • Encrypt and decrypt records
      • Configuring an Encrypt and Decrypt Field processor
    • Expression Evaluator
      • Output fields and attributes
      • Record header attribute expressions
      • Field attribute expressions
      • Configuring an Expression Evaluator processor
    • Field Flattener
      • Flatten the entire record
      • Flatten specific fields
      • Configuring a Field Flattener processor
    • Field Hasher
      • Hash methods
      • Field separator
      • List, map, and list-map fields
      • Configuring a Field Hasher processor
    • Field Mapper
      • Field values
      • Field names
      • Field paths
        • Aggregation expressions
      • Configuring a Field Mapper processor
    • Field Masker
      • Mask types
      • Configuring a Field Masker processor
    • Field Merger
      • Configuring a Field Merger processor
    • Field Order
      • Missing and extra fields
      • Configuring a Field Order processor
    • Field Pivoter
      • Generated records
      • Configuring a Field Pivoter processor
    • Field Remover
      • Configuring a Field Remover processor
    • Field Renamer
      • Renaming sets of fields
      • Configuring a Field Renamer processor
    • Field Replacer
      • Replacing values with nulls
      • Replacing values with new values
      • Data types for conditional replacement
      • Configuring a Field Replacer processor
    • Field Splitter
      • Not enough splits
      • Too many splits
      • Example
      • Configuring a Field Splitter processor
    • Field Type Converter
      • Valid type conversions
      • Expressions to convert by field name
      • Changing the scale of decimal fields
      • Configuring a Field Type Converter processor
    • Field Zip
      • Merging List data
      • Merging List-Map data
      • Pivoting merged lists
      • Configuring a Field Zip processor
    • Geo IP
      • Supported databases
      • Database file location
      • GeoIP field types
        • Full JSON field types
      • Configuring a Geo IP processor
    • Groovy Evaluator
      • Processing mode
      • Groovy scripting objects
      • Accessing record details
      • Processing list-map data
      • Type handling
      • Event generation
      • Working with record header attributes
        • Viewing record header attributes
      • Accessing whole file format records
      • Calling external Java code
      • Granting permissions on Groovy scripts
      • Configuring a Groovy Evaluator processor
    • HBase Lookup
      • Lookup key
      • Lookup cache
      • Kerberos authentication
      • Using an HBase user
      • HDFS properties and configuration file
      • Configuring an HBase Lookup processor
    • Hive Metadata
      • Output streams
      • Metadata records and record header attributes
        • Custom record header attributes
      • Database, table, and partition expressions
        • Hive names and supported characters
      • Decimal field expressions
      • Time basis
      • Cache
        • Cache size and evictions
      • Kerberos authentication
      • Hive properties and configuration files
      • Hive data types
      • Configuring a Hive Metadata processor
    • HTTP Client
      • HTTP method
        • Expression method
      • Headers
      • Per-status actions
      • Pass records after retry or timeout failures
        • Record header attributes for passed records
      • Pagination
        • Page or offset number
        • Result field path
        • Keep all fields
        • Pagination examples
          • Example for link in HTTP header
          • Example for link in response field
          • Example for page number
          • Example for offset number
      • OAuth 2 authorization
        • Example for Twitter
        • Example for Microsoft Azure AD
        • Example for Google
      • Generated output
        • Response headers
      • Logging request and response data
      • Logging the resolved resource URL
      • Data formats
      • Configuring an HTTP Client processor
    • HTTP Router
      • Configuring an HTTP Router processor
    • JavaScript Evaluator
      • Processing mode
      • JavaScript scripting objects
      • Accessing record details
      • Processing list-map data
      • Type handling
      • Event generation
      • Working with record header attributes
        • Viewing record header attributes
      • Accessing whole file format records
      • Calling external Java code
      • Configuring a JavaScript Evaluator processor
    • JDBC Lookup
      • Database vendors and drivers
        • MySQL data types
        • Oracle data types
        • PostgreSQL data types
        • SQL Server data types
        • Unsupported data types
      • Installing the JDBC driver
      • Lookup cache
        • Using additional threads
        • Retry lookups for missing values
      • JDBC field attributes
      • Configuring a JDBC Lookup processor
    • JDBC Tee
      • Database vendors and drivers
        • MySQL data types
        • PostgreSQL data types
      • Installing the JDBC driver
      • CRUD operation processing
      • Single and multi-row operations
      • Configuring a JDBC Tee processor
    • JSON Generator
      • Configuring a JSON Generator processor
    • JSON Parser
      • Configuring a JSON Parser processor
    • Jython Evaluator
      • Processing mode
      • Jython scripting objects
      • Accessing record details
      • Processing list-map data
      • Type handling
      • Event generation
      • Working with record header attributes
        • Viewing Record Header Attributes
      • Accessing whole file format records
      • Calling external Java code
      • Calling an external Python module
      • Configuring a Jython Evaluator processor
    • Kaitai Struct Parser
      • Example
      • Prerequisite task
      • Configuring the Kaitai Struct Parser processor
    • Kudu Lookup
      • Column mappings
      • Kudu data types
      • Lookup cache
        • Cache table information
        • Cache lookup values
      • Kerberos authentication
      • Configuring a Kudu Lookup processor
    • Log Parser
      • Log formats
      • Configuring a Log Parser processor
    • MLeap Evaluator
      • Prerequisites
      • MLeap model as a microservice
      • Example: Airbnb model
      • Configuring an MLeap Evaluator processor
    • MongoDB Atlas Lookup
      • Field mappings
      • Lookup cache
      • Credentials
      • Read preference
      • Configuring a MongoDB Atlas Lookup processor
    • PMML Evaluator
      • Prerequisites
        • Installing the PMML stage library
      • PMML model as a microservice
      • Example: Iris classification
      • Configuring a PMML Evaluator processor
    • PostgreSQL Metadata
      • JDBC driver
      • Schema and table names
      • Decimal precision and scale field attributes
      • Caching information
      • PostgreSQL data types
      • Configuring a PostgreSQL Metadata processor
    • Record Deduplicator
      • Comparison window
      • Configuring a Record Deduplicator processor
    • Redis Lookup
      • Data types
      • Lookup cache
      • Configuring a Redis Lookup processor
    • Salesforce Bulk API 2.0 Lookup
      • Lookup cache
      • Salesforce attributes
      • Changing the API version
      • Configuring a Salesforce Bulk API 2.0 Lookup processor
    • Salesforce Lookup
      • Lookup mode
        • Aggregate functions in SOQL queries
      • Lookup cache
      • Salesforce attributes
      • Changing the API version
      • Configuring a Salesforce Lookup processor
    • Schema Generator
      • Using schema header attributes
      • Generated schemas
      • Caching schemas
      • Configuring a Schema Generator processor
    • SQL Parser
      • Using multiple pipelines
        • Example
      • Resolving the schema
      • Unsupported data types
      • Generated records
        • CRUD operation header attributes
        • CDC header attributes
        • Other header attributes
        • Field attributes
      • Configuring an SQL Parser processor
    • Start Jobs
      • Job execution and data flow
      • Generated record
      • Suffix for job instance names
      • Runtime parameters for jobs
      • Configuring a Start Jobs processor
    • Static Lookup
      • Configuring a Static Lookup processor
    • Stream Selector
      • Default stream
      • Sample conditions for streams
      • Configuring a Stream Selector processor
    • TensorFlow Evaluator
      • Prerequisites
      • Evaluation method
        • Example: Each record
      • Event generation
        • Event records
      • Serving a TensorFlow model
      • Configuring a TensorFlow Evaluator processor
    • Wait for Jobs
      • Stage processing and pipeline implementation
      • Generated record
      • Configuring a Wait for Jobs processor
    • Web Client
      • Comparing Web Client and HTTP Client processors
      • HTTP method
        • Expression method
      • Headers
      • Grouping style
      • Event generation
        • Event records
      • Per-status actions
      • Per-timeout actions
      • Pagination
        • Page or offset number
      • OAuth 2 authentication
      • Generated output
      • Data formats
      • Configuring a Web Client processor
    • Whole File Transformer
      • Implementation overview
        • Write Avro pipeline
        • Parquet conversion pipeline
      • Memory and storage requirements
      • Generated record
      • Implementation example: Amazon S3 Parquet conversion
        • Write Avro pipeline example
        • Parquet conversion pipeline example
        • Runtime
      • Configuring a Whole File Transformer processor
    • Windowing Aggregator
      • Window type, time windows, and information display
        • Rolling windows
        • Sliding windows
      • Calculation components
      • Aggregate functions
      • Event generation
        • Event record root field
        • Event records
      • Monitoring aggregations
      • Configuring a Windowing Aggregator processor
    • XML Flattener
      • Generated records
      • Configuring an XML Flattener processor
    • XML Parser
      • Configuring an XML Parser processor
  • Destinations
    • Destinations
    • Amazon S3
      • Authentication method
      • Bucket
      • Partition prefix
      • Time basis and data time zone for time-based buckets and partition prefixes
      • Object names
        • Whole file names
      • Add tags to objects
      • Event generation
        • Event records
      • Server-side encryption
      • Data formats
      • Configuring an Amazon S3 destination
    • Apache Iceberg for AWS
      • AWS Prerequisites
      • Creating the Iceberg table
        • Partitioning a new table
      • Write mode
      • Schema evolution mode
        • Supported type promotions
      • Amazon server-side encryption
      • Processing timestamp data
      • Iceberg data types
      • Configuring an Apache Iceberg for AWS destination
    • Azure Blob Storage
      • Prerequisite
      • Blob types
      • Event generation
        • Event records
      • Data formats
      • Configuring an Azure Blob Storage destination
    • Azure Data Lake Storage Gen2
      • Prerequisites
        • Retrieve authentication information
      • Directory templates
      • Time basis
      • Late records and late record handling
      • Timeout to close idle files
      • Recovery
      • Event generation
        • Event records
      • Resolving out of memory errors
      • Data formats
      • Configuring an Azure Data Lake Storage Gen2 destination
    • Azure Event Hub Producer
      • Data formats
      • Configuring an Azure Event Hub Producer destination
    • Azure IoT Hub Producer
      • Register Data Collector as an IoT Hub device
      • Data formats
      • Configuring an Azure IoT Hub Producer destination
    • Azure Synapse SQL
      • Prerequisites
        • Prepare the Azure Synapse instance
        • Prepare the Azure Storage staging area
        • Enable access to the container
      • Load methods
      • Connections and authentication
        • Azure Synapse connection
        • Staging connection
        • Copy statement connection
      • Specifying tables
      • Enabling data drift handling
      • Row generation
        • Missing fields and fields with invalid types
      • Performance optimization
      • Azure Synapse data types
      • CRUD operation processing
      • Configuring an Azure Synapse SQL destination
    • Cassandra
      • Batch type
      • Authentication
        • Kerberos (DSE) authentication
      • Cassandra data types
      • Configuring a Cassandra destination
    • CoAP Client
      • Data formats
      • Configuring a CoAP Client destination
    • Couchbase
      • CRUD operation processing
      • Compare and swap
      • Data formats
      • Configuring a Couchbase destination
    • Databricks
      • Prerequisite
      • Load methods
        • CRUD operation processing
        • Primary key location
      • Specifying tables
      • Enabling data drift handling
      • Partitioning tables
      • Performance optimization
      • Staging location
        • Amazon S3 credentials
        • ADLS Gen2 authentication information
        • Google Cloud credentials
      • Row generation
        • Missing fields and fields with invalid types
      • Databricks data types
      • Configuring a Databricks destination
    • Elasticsearch
      • Security
      • Time basis and time-based indexes
      • Document IDs
      • CRUD operation processing
      • Configuring an Elasticsearch destination
    • Google BigQuery
      • Prerequisites
        • Prepare the Google BigQuery data warehouse
        • Prepare the Google Cloud Storage staging area
      • Credentials
      • Staging file formats
      • Specifying datasets and tables
      • Enabling data drift handling
        • Partitioning new tables
          • Partition configurations
          • Partition options
      • Row generation
        • Missing fields and fields with invalid types
      • Performance optimization
      • CRUD operation processing
      • BigQuery data types
      • Configuring a Google BigQuery destination
    • Google Bigtable
      • Prerequisites
        • Install the BoringSSL library
        • Configure credentials
      • Row key
      • Cloud Bigtable data types
      • Column family and field mappings
      • Time basis
      • Configuring a Google Bigtable destination
    • Google Cloud Storage
      • Credentials
      • Partition prefix
      • Time basis, data time zone, and time-based partition prefixes
      • Object names
        • Whole file names
      • Event generation
        • Event records
      • Data formats
      • Configuring a Google Cloud Storage destination
    • Google Pub/Sub Publisher
      • Credentials
      • Data formats
      • Configuring a Google Pub/Sub Publisher destination
    • Hadoop FS
      • Directory templates
      • Time basis
      • Late records and late record handling
      • Timeout to close idle files
      • Recovery
      • Data formats
      • Writing to Azure Blob storage
      • Writing to Azure Blob Storage with HDInsight
      • Event generation
        • Event records
      • Kerberos authentication
      • Impersonation user
      • HDFS properties and configuration files
      • Configuring a Hadoop FS destination
    • HBase
      • Field mappings
      • Kerberos authentication
      • Using an HBase user
      • Time basis
      • HDFS properties and configuration file
      • Configuring an HBase destination
    • Hive Metastore
      • Metadata processing
      • Hive table generation
      • Cache
        • Cache size and evictions
      • Event generation
        • Event records
      • Kerberos authentication
      • Hive properties and configuration files
      • Hive data types
      • Configuring a Hive Metastore destination
    • HTTP Client
      • HTTP method
        • Expression method
      • Headers
      • Number of requests
      • Logging request and response data
      • Send microservice responses
      • OAuth 2 authorization
        • Example for Twitter
        • Example for Microsoft Azure AD
        • Example for Google
      • Data formats
      • Configuring an HTTP Client destination
    • IBM Cloud Object Storage
      • Prerequisites
      • Bucket
      • Troubleshooting IBM Connectivity Service
      • Configuring an IBM Cloud Object Storage destination
    • IBM Db2
      • Prerequisites
      • Specifying tables
      • Key columns
      • IBM Db2 data types
      • Troubleshooting IBM Connectivity Service
      • Configuring an IBM Db2 destination
    • IBM watsonx.data
      • Prerequisites
      • Specifying tables
        • Creating tables
      • watsonx.data data types
      • Troubleshooting IBM Connectivity Service
      • Configuring an IBM watsonx.data destination
    • InfluxDB 2.x
      • Configuring an InfluxDB 2.x destination
    • JDBC Producer
      • Database vendors and drivers
      • Installing the JDBC driver
      • CRUD operation processing
      • Update and delete operations
      • Single and multi-row operations
      • Configuring a JDBC Producer destination
    • Jira
      • Event generation
        • Event records
      • OAuth 2 authentication
      • Configuring a Jira destination
    • JMS Producer
      • Installing JMS drivers
      • Include headers
      • Additional JMS properties
      • Working with TIBCO EMS
        • Configuring for TIBCO and SSL
      • Data formats
      • Configuring a JMS Producer destination
    • Kafka Producer
      • Broker list
      • Runtime topic resolution
      • Partition strategy
      • Send microservice responses
      • Additional Kafka properties
      • Writing Kafka message headers
      • Kafka security
      • Data formats
      • Configuring a Kafka Producer destination
    • Kinesis Firehose
      • Authentication method
      • Delivery stream
      • Data formats
      • Configuring a Kinesis Firehose destination
    • Kinesis Producer
      • Authentication method
      • Additional Kinesis properties
      • Send Microservice responses
      • Data formats
      • Configuring a Kinesis Producer destination
    • Kudu
      • CRUD operation processing
      • Kudu data types
      • Kerberos authentication
      • Configuring a Kudu destination
    • Local FS
      • Directory templates
      • Time basis
      • Late records and late record handling
      • Timeout to close idle files
      • Recovery
      • Event generation
        • Event records
      • Data formats
      • Configuring a Local FS destination
    • MapR DB
      • Field mappings
      • Time basis
      • Kerberos authentication
      • Using an HBase user
      • HDFS properties and configuration file
      • Configuring a MapR DB destination
    • MapR DB JSON
      • Row key
        • Row key data types
      • Writing to MapR DB JSON
        • Define the CRUD Operation
        • Insert and Set API Properties
      • Configuring a MapR DB JSON destination
    • MapR FS
      • Directory templates
      • Time basis
      • Late records and late record handling
      • Timeout to close idle files
      • Recovery
      • Event generation
        • Event records
      • Data formats
      • Kerberos authentication
      • Impersonation user
      • HDFS properties and configuration files
      • Configuring a MapR FS destination
    • MapR Streams Producer
      • Data formats
      • Runtime topic resolution
      • Partition strategy
      • Additional properties
      • Configuring a MapR Streams Producer destination
    • MongoDB Atlas
      • Credentials
      • Specifying field paths
      • Unordered writes and stopping the pipeline
      • Define the CRUD operation
        • Performing upserts
      • Enabling SSL/TLS
      • MongoDB data types
      • Writing BSON types
      • Configuring a MongoDB Atlas destination
    • MQTT Publisher
      • Topic
      • Data formats
      • Configuring an MQTT Publisher destination
    • Named Pipe
      • Prerequisites
        • Create the named pipe
        • Configure the named pipe reader
      • Working with the named pipe reader
      • Data formats
      • Configuring a named pipe destination
    • Oracle
      • Prerequisite
      • Enabling data drift handling
        • Generated data types
      • Creating tables
      • CRUD operation processing
      • Configuring an Oracle destination
    • Pulsar Producer
      • Schema properties
      • Security
        • Enabling TLS transport encryption
        • Enabling TLS authentication
        • Enabling JWT authentication
        • Enabling OAuth authentication
      • Data formats
      • Configuring a Pulsar Producer destination
    • RabbitMQ Producer
      • Queue
      • Data formats
      • Configuring a RabbitMQ Producer destination
    • Redis
      • Mode
        • Data types for batch mode
        • Data formats for publish mode
      • Define the CRUD operation
      • Configuring a Redis destination
    • Salesforce
      • CRUD operation processing
      • Hard deleting records
      • Field mappings
      • Changing the API version
      • Configuring a Salesforce destination
    • Salesforce Bulk API 2.0
      • CRUD operation processing
      • Hard deleting records
      • Field mappings
      • Changing the API version
      • Configuring a Salesforce Bulk API 2.0 Destination
    • Send Response to Origin
      • Configuring a Send Response to Origin destination
    • SFTP/FTP/FTPS Client
      • Credentials
      • Event generation
        • Event records
      • Data format
      • Configuring an SFTP/FTP/FTPS Client destination
    • SingleStore
      • CRUD operation processing
      • Configuring a SingleStore destination
    • Snowflake
      • Sample use cases
      • Prerequisites
        • Create a Snowflake stage
          • AWS credentials
          • Google Cloud Storage credentials
        • DESCRIBE prerequisite
        • COPY prerequisites
        • Snowpipe prerequisites
        • MERGE prerequisites
      • Implementation notes
      • Load methods
        • Primary key location
      • Error handling
      • Define a role
      • Performance optimization
      • Row generation
      • Specifying tables
      • Enabling data drift handling
        • Generated data types
      • Creating tables
      • CRUD operation processing
      • Configuring a Snowflake destination
    • Snowflake File Uploader
      • Prerequisites
        • Create an internal Snowflake stage (optional)
        • Assign required privileges
      • Implementation notes
      • Define a role
      • Event generation
        • Event records
      • Configuring a Snowflake File Uploader destination
    • Solr
      • Index mode
      • Kerberos authentication
      • Configuring a Solr destination
    • Splunk
      • Prerequisites
      • Required record format
      • Logging request and response data
      • Configuring a Splunk destination
    • Syslog
      • Protocol
      • Using SSL/TLS
      • Enabling SSL
      • Message content
      • Data formats
      • Configuring a Syslog destination
    • Tableau CRM
      • Changing the API version
      • Define the operation
        • Metadata JSON
      • Automatic recovery
        • Manual upload recovery
      • Configuring an Tableau CRM destination
    • Teradata
      • Prerequisites
        • Installing the Teradata driver
        • COPY prerequisites
        • MERGE prerequisites
      • Load methods
        • Primary key location
      • Performance optimization
      • Specifying tables
      • Enabling data drift handling
        • Generated data types
      • Creating tables
      • Staging location
        • Amazon S3 credentials
        • Azure authentication information
        • Google Cloud credentials
      • Row generation
        • Missing fields and fields with invalid types
      • CRUD operation processing
      • Configuring a Teradata destination
    • To Error
    • Trash
    • Web Client
      • Comparing Web Client and HTTP Client destinations
      • HTTP method
        • Expression method
      • Headers
      • Grouping style
      • Event generation
        • Event records
      • Per-status actions
      • Per-timeout actions
      • Pagination
        • Page or offset number
      • OAuth 2 authentication
      • Data formats
      • Configuring a Web Client destination
    • WebSocket Client
      • Data formats
      • Configuring a WebSocket Client destination
  • Executors
    • Executors
    • ADLS Gen2 File Metadata
      • Prerequisites
        • Retrieve authentication information
      • Related event generating stages
      • Changing metadata
        • Specifying the file path
        • Changing the file name or location
        • Defining the owner, group, permissions, and ACLs
      • Creating an empty file
      • Removing a file or directory
      • Event generation
        • Event records
      • Configuring an ADLS Gen2 File Metadata executor
    • Amazon S3
      • Authentication method
      • Create new objects
      • Copy objects
      • Tag existing objects
      • Event generation
        • Event records
      • Server-side encryption
      • Configuring an Amazon S3 executor
    • Databricks Job Launcher
      • Prerequisites
      • Event generation
        • Event records
      • Monitoring
      • Configuring a Databricks Job Launcher executor
    • Databricks Query
      • Prerequisite
      • Spark SQL queries
      • Storage connection
        • ADLS Gen2 authentication information
      • Event generation
        • Event records
      • Configuring a Databricks Query executor
    • Email
      • Prerequisite
      • Conditions
      • Using expressions
      • Configuring an Email executor
    • Google Cloud Storage
      • Credentials
      • Create new objects
      • Copy or move objects
      • Set metadata
      • Event generation
        • Event records
      • Configuring a Google Cloud Storage executor
    • Google BigQuery
      • Prerequisite
      • Credentials
      • SQL queries
      • Event generation
        • Event records
      • Configuring a Google BigQuery executor
    • HDFS File Metadata
      • Related event generating stages
      • Changing metadata
        • Specifying the file path
        • Changing the file name or location
        • Defining the owner, group, permissions, and ACLs
      • Creating an empty file
      • Removing a file or directory
      • Event generation
        • Event records
      • Kerberos authentication
      • HDFS user
      • HDFS properties and configuration files
      • Configuring an HDFS File Metadata executor
    • Hive Query
      • Related event generating stages
      • Installing the Impala driver
      • Hive and Impala queries
      • Impala queries for the drift synchronization solution for Hive
      • Event generation
        • Event records
      • Configuring a Hive Query executor
    • JDBC Query
      • Database vendors and drivers
      • Installing the JDBC driver
      • SQL queries
      • Event generation
        • Event records
      • Configuring a JDBC Query executor
    • MapR FS File Metadata
      • Related event generating stage
      • Changing metadata
        • Specifying the file path
        • Changing the file name or location
        • Defining the owner, group, permissions, and ACLs
      • Creating an empty file
      • Removing a file or directory
      • Event generation
        • Event records
      • Kerberos authentication
      • HDFS user
      • HDFS properties and configuration files
      • Configuring a MapR FS File Metadata executor
    • MapReduce
      • Prerequisites
      • Related event generating stages
      • MapReduce jobs and job configuration properties
        • Predefined jobs for Parquet and ORC
      • Event generation
        • Event records
      • Kerberos authentication
      • Using a MapReduce user
      • Configuring a MapReduce executor
    • Pipeline Finisher
      • Recommended implementation
      • Related event generating stages
      • Origin reset for additional pipeline runs
      • Notification options
      • Configuring a Pipeline Finisher executor
    • SFTP/FTP/FTPS Client
      • Credentials
      • Configuring an SFTP/FTP/FTPS Client executor
    • Shell
      • Data Collector shell impersonation mode
        • Using a Partial Control Hub User ID
      • Script configuration
      • Configuring a Shell executor
    • Snowflake
      • Prerequisite
      • Implementation notes
        • Snowflake File Uploader and executor pipelines
      • Define a role
      • SQL queries
      • Event generation
        • Event record
      • Configuring a Snowflake executor
    • Spark
      • Spark versions and stage libraries
      • Prerequisite
      • Spark home requirement
      • Application properties
      • Using a proxy Hadoop user
      • Kerberos authentication
      • Event generation
        • Event records
      • Monitoring
      • Configuring a Spark executor
  • Dataflow triggers
    • Dataflow triggers overview
    • Pipeline event generation
      • Using pipeline events
        • Pass to an executor
    • Stage event generation
      • Using stage events
        • Task execution streams
        • Event storage streams
    • Executors
    • Logical pairings
    • Event records
      • Event record header attributes
    • Viewing events in data preview and snapshot
      • Viewing stage events in data preview and snapshot
    • Executing pipeline events in data preview
    • Summary
  • Solutions
    • Solutions overview
    • Converting data to the Parquet data format
    • Automating Impala metadata updates for drift synchronization for Hive
    • Managing output files
    • Stopping a pipeline after processing all available data
    • Offloading data from relational sources to Hadoop
    • Sending email during pipeline processing
    • Preserving an audit trail of events
    • Loading data into Databricks Delta Lake
      • Bulk loading data into a Delta Lake table
        • Create the pipeline and configure the Salesforce origin
        • Configure the Expression Evaluator processor
        • Configure the destination to bulk load data
        • Run the pipeline to bulk load data
      • Merging changed data into a Delta Lake table
        • Create the pipeline and configure the MySQL Binary Log origin
        • Configure processors to restructure the record
        • Configure the destination to merge changed data
        • Run the pipeline to merge changed data
    • Drift Synchronization Solution for Hive
      • General processing
        • Parquet processing
        • Impala support
        • Flatten records
      • Basic Avro implementation
      • Basic Parquet implementation
      • Implementation steps
      • Avro case study
        • Hive Metadata processor for Avro data format
        • Hive Metastore for Avro data format
        • Data-processing destination for Avro data format
        • Processing Avro data
      • Parquet case study
        • JDBC Query Consumer origin
        • Hive Metadata processor for Parquet data format
        • Hive Metastore destination for Parquet data format
        • Data-processing destination for Parquet data format
        • MapReduce executor
        • Processing Parquet data
      • Hive data types
    • Drift Synchronization Solution for PostgreSQL
      • Basic implementation and processing
        • Flattening records
      • Requirements
      • Implementation steps
      • Case study
        • JDBC Multitable Consumer origin
        • PostgreSQL Metadata processor
        • JDBC Producer destination
        • Running the pipeline
      • PostgreSQL data types
  • Multithreaded pipelines
    • Multithreaded pipeline overview
    • How it works
      • Origins for multithreaded pipelines
      • Processor caching
    • Monitoring
    • Tuning threads and runners
    • Resource usage
    • Multithreaded pipeline summary
  • Microservice pipelines
    • Microservice pipelines overview
    • Stages for microservice pipelines
      • Microservice origins
      • Microservice destinations
    • Sample pipeline
    • Creating a microservice pipeline
  • Orchestration pipelines
    • Orchestration pipeline overview
    • Orchestration stages
      • Retrying job requests
    • Orchestration record
      • Using the orchestration record
    • Sample pipeline
  • Data Preview
    • Data Preview Overview
      • Data Preview Availability
      • Source Data for Data Preview
      • Writing to Destinations and Executors
      • Notes
    • Preview Codes
    • Input and Output Schema for Stages
    • Previewing a Single Stage
    • Previewing Multiple Stages
    • Editing Preview Data
    • Editing Properties
  • Rules and Alerts
    • Rules and Alerts Overview
    • Metric Rules and Alerts
      • Default Metric Rules
      • Metric Types
        • Gauge
        • Counter
        • Histogram
        • Meter
        • Timer
      • Metric Conditions
      • Configuring a Metric Rule and Alert
    • Data Rules and Alerts
      • Configuring a Data Rule and Alert
    • Data Drift Rules and Alerts
      • Data Drift Alert Triggers
      • Configuring Data Drift Rules and Alerts
    • Alert Webhooks
      • Configuring an Alert Webhook
    • Configuring Email for Alerts
  • Troubleshooting
    • Accessing error messages
    • Pipeline basics
      • Data preview
      • General validation errors
    • Origins
      • Directory
      • Elasticsearch
      • JDBC origins
      • Oracle CDC Client
      • PostgreSQL CDC Client
      • Salesforce
      • Scripting origins
      • SQL Server CDC Client
    • Processors
      • Encrypt and Decrypt Fields
    • Destinations
      • Azure Data Lake Storage
      • Cassandra
      • Elasticsearch destination
      • Hadoop FS
      • HBase
      • Kafka Producer
    • Executors
      • Hive Query
    • JDBC connections
      • No suitable driver
      • Cannot connect to database
      • MySQL JDBC driver and time values
    • Performance
  • Data formats by stage
    • Data format support
    • Origins
    • Processors
    • Destinations
  • Expression language
    • Expression language
      • Expression completion in properties
        • Tips for Expression Completion
      • Expression examples
    • Functions
      • Record functions
      • Delimited data record functions
      • Error record functions
      • Base64 functions
      • Credential functions
      • Data drift functions
      • Data generation functions
        • Faker functions
        • Xeger functions
      • Field functions
      • File functions
      • Job functions
      • Math functions
      • Pipeline functions
      • String functions
      • Time functions
      • Timestamp functions
      • Miscellaneous functions
    • Constants
    • Datetime variables
    • Literals
    • Operators
      • Operator precedence
    • Reserved words
  • Regular expressions
    • Regular expressions overview
    • Regular expressions in the pipeline
    • Quick reference
    • Regex examples
  • Grok patterns
    • Defining grok patterns
    • General grok patterns
    • Date and time grok patterns
    • Java grok patterns
    • Log grok patterns
    • Networking grok patterns
    • Path grok patterns
• Pipeline Design (Transformer)
  • Getting Started
    • What is Transformer?
    • Pipeline Processing on Spark
    • Batch Case Study
    • Streaming Case Study
    • Transformer for Data Collector Users
    • Tutorials and Sample Pipelines
  • Cluster Types
    • Overview
      • Spark Job Reporting
    • Amazon EMR
      • Transformer Installation Location
      • Authentication Method
      • Server-Side Encryption
        • KMS Requirements
      • Kerberos Stage Limitations
      • Force Stopping an EMR Job
      • Existing Cluster
        • Specifying a Cluster
      • Transformer-Provisioned Cluster
        • Bootstrap Actions
        • Transformer-Provisioned Cluster Names
      • AWS-Provisioned Cluster
        • AWS Prerequisites
        • AWS-Provisioned Cluster Names
      • S3 Staging URI and Directory
      • Monitoring EMR Pipelines
    • Amazon EMR Serverless
      • VPC Access
      • Authentication Method
      • Existing Application
        • Specifying an Application
      • Transformer-Provisioned Application
      • AWS-Provisioned Application
        • AWS Prerequisites
        • AWS-Provisioned Application Names
      • Staging Location
      • Monitoring EMR Serverless Pipelines
    • Cloudera Data Engineering
      • Supported Stages
      • Job Resource and Resource File Prefix
      • Amazon S3 and HDFS Prerequisites
    • Databricks
      • Spark Properties for Google Stages
      • Existing Cluster
        • Uninstalling Transformer Libraries
      • Provisioned Cluster
        • Cluster-Scoped Init Scripts
          • Access Keys for ABFSS Init Scripts
        • Cluster Configuration
        • Using an Instance Pool
        • Locating Properties in Databricks
      • Staging Directory
        • Staging Data on Unity Catalog
        • External Resources
          • Caching External Resource Files
        • Limiting Staging Directory Access
      • Retrying Pipelines
      • Accessing Databricks Job Details
    • Google Dataproc
      • Supported Stages
      • Credentials
        • Default Credentials
        • Credentials in a File
        • Credentials in a Property
      • Existing Cluster
      • Provisioned Cluster
      • GCS Staging URI
      • Accessing Dataproc Job Details
    • Hadoop YARN
      • Deployment Mode
      • Transformer Proxy Users
      • Kerberos Authentication
        • Using a Keytab for Each Pipeline
      • Hadoop Impersonation Mode
        • Lowercasing User Names
        • Working with HDFS Encryption Zones
  • Pipeline Design
    • What is a Transformer Pipeline?
    • Sample Pipelines
    • Stage Library Match Requirement
    • Local Pipelines
    • Spark Executors
    • Partitioning
      • Initial Partitions
      • Shuffling
      • Repartitioning
    • Offset Handling
      • Skip Offset Tracking
      • Reset Pipeline Offsets
    • Batch Header Attributes
    • Delivery Guarantee
    • Performing Lookups
      • Streaming Lookup Example
    • Expressions in Pipeline and Stage Properties
      • Referencing Fields in Spark SQL Expressions
    • Data Types
      • Data Types in Preview
    • Deprecated Functionality
  • Pipeline Configuration
    • Execution Mode
    • Cluster Callback URL
    • Preprocessing Script
      • Preprocessing Script Requirements
    • Extra Spark Configuration
      • Performance Tuning Properties
    • Ludicrous Processing Mode
      • Caching Support
      • Statistics
    • Runtime Values
      • Runtime Parameters
        • Step 1. Define a Runtime Parameter
        • Step 2. Call the Runtime Parameter
          • Calling from Text Boxes
          • Calling from Checkboxes and Drop-Down Menus
          • Calling from Scripting Processors
      • Runtime Properties
        • Step 1. Define Runtime Properties
        • Step 2. Call the Runtime Property
      • Runtime Resources
        • Step 1. Define Runtime Resources
        • Step 2. Call Runtime Resources
    • Simple and Bulk Edit Mode
    • Caching Data
    • Cache Levels and Replicas
    • Validation
      • Spark for Explicit Validation
    • Amazon Security
      • Assume Another Role
        • Stage Library and Cluster Type Requirements
        • Assume Role Methods
        • Create the Trust Policy
        • Configure Transformer to Assume a Role
    • Security in Kafka Stages
      • Prerequisite Tasks
      • SASL Authentication Credentials
        • Providing PLAIN Credentials
        • Providing Kerberos Credentials
      • Enabling SSL/TLS Encryption
      • Enabling SSL/TLS Encryption and Authentication
      • Enabling SASL Authentication
      • Enabling SASL Authentication on SSL/TLS
      • Enabling Custom Authentication
    • Kafka Message Keys
    • Configuring a Pipeline
  • Origins
    • Overview
    • Schema Inference
    • Custom Schemas
      • Schema Application
      • Error Handling
      • DDL Schema Format
      • JSON Schema Format
    • ADLS Gen2
      • Prerequisites
        • Retrieve Authentication Information
      • Schema Requirement
      • Partitioning
      • Data Formats
      • Configuring an ADLS Gen2 Origin
    • Amazon S3
      • Prerequisites
      • Authentication Method
      • Partitioning
      • Data Formats
      • Configuring an Amazon S3 Origin
    • Azure Event Hubs
      • Prerequisites
      • Default and Specific Offsets
      • Configuring an Azure Event Hubs Origin
    • Delta Lake
      • Storage Systems
      • ADLS Gen2 Prerequisites
        • Retrieve Authentication Information
      • Amazon S3 Credential Mode
      • Reading from a Local File System
      • Configuring a Delta Lake Origin
    • File
      • Schema Requirement
      • Directory Path
      • Partitioning
      • Data Formats
      • Configuring a File Origin
    • Google Big Query
      • Incremental and Full Query Mode
      • Offset Column and Supported Types
      • BigQuery Data Types
      • Configuring a Big Query Origin
    • JDBC Query
      • Database Vendors and Drivers
      • Installing the JDBC Driver
      • SQL Queries
      • Configuring a JDBC Query Origin
    • JDBC Table
      • Database Vendors and Drivers
      • Installing the JDBC Driver
      • Partitioning
      • Offset Column Requirement
        • Supported Offset Data Types
      • Configuring a JDBC Table Origin
    • Hive
      • Partitioning
      • Reading a Delta Lake Managed Table
      • Incremental and Full Query Mode
        • SQL Query
          • Incremental Mode Guidelines
          • Full Mode Guidelines
      • Additional Hive Configuration Properties
      • Configuring a Hive Origin
    • Kafka
      • Partitioning
      • Topic Specification
      • Offsets
      • Kafka Security
      • Data Formats
      • Configuring a Kafka Origin
    • Kudu
      • Configuring a Kudu Origin
    • MySQL JDBC Table
      • Installing the MySQL JDBC Driver
      • Offset Column
        • Supported Offset Data Types
        • Null Offset Value Handling
      • Default Offset Queries
      • Custom Offset Queries
      • Partitioning
        • Partition Column Selection
      • MySQL Data Types
      • Configuring a MySQL JDBC Table Origin
    • Oracle JDBC Table
      • Installing the Oracle JDBC Driver
      • Offset Column
        • Supported Offset Data Types
        • Null Offset Value Handling
      • Default Offset Queries
      • Custom Offset Queries
      • Partitioning
        • Partition Column Selection
      • Oracle Data Types
      • Configuring an Oracle JDBC Table Origin
    • PostgreSQL JDBC Table
      • PostgreSQL JDBC Driver
      • Offset Column
        • Supported Offset Data Types
        • Null Offset Value Handling
      • Default Offset Queries
      • Custom Offset Queries
      • Partitioning
        • Partition Column Selection
      • PostgreSQL Data Types
      • Configuring a PostgreSQL JDBC Table Origin
    • Snowflake
      • Required Privileges and Custom Roles
      • Read Mode
      • Full or Incremental Read
      • SQL Query Guidelines
        • Incremental Read Query
        • Full Read Query
      • Pushdown Optimization
      • Snowflake Data Types
      • Configuring a Snowflake Origin
    • SQL Server JDBC Table
      • SQL Server JDBC Driver
      • Offset Column
        • Supported Offset Data Types
        • Null Offset Value Handling
      • Default Offset Queries
      • Custom Offset Queries
      • Partitioning
        • Partition Column Selection
      • SQL Server Data Types
      • Configuring a SQL Server JDBC Table Origin
    • Unity Catalog
      • Full or Incremental Read
      • Configuring a Unity Catalog Origin
    • Whole Directory
      • Data Formats
      • Configuring a Whole Directory Origin
  • Processors
    • Overview
    • Aggregate
      • Aggregate Processing
      • Aggregate Functions
      • Default Output Fields
      • Configuring an Aggregate Processor
    • Deduplicate
      • Configuring a Deduplicate Processor
    • Delta Lake Lookup
      • Storage Systems
      • ADLS Gen2 Prerequisites
        • Retrieve Authentication Information
      • Amazon S3 Credential Mode
      • Using a Local File System
      • Configuring a Delta Lake Lookup Processor
    • Field Flattener
      • Flatten all Fields
      • Flatten Specific Fields
      • Configuring a Field Flattener Processor
    • Field Order
      • Configuring a Field Order Processor
    • Field Remover
      • Match Criteria
      • Configuring a Field Remover Processor
    • Field Renamer
      • Rename Methods
      • Configuring a Field Renamer Processor
    • Field Replacer
      • Replacing Values with Nulls
      • Replacing Values with New Values
      • Configuring a Field Replacer Processor
    • Filter
      • Filter Condition
      • Configuring a Filter Processor
    • JDBC Lookup
      • Database Vendors and Drivers
      • Installing the JDBC Driver
      • Configuring a JDBC Lookup Processor
    • Join
      • Join Types
        • Cross Join
        • Inner Join
        • Full Outer Join
        • Left Anti Join
        • Left Outer Join
        • Left Semi Join
        • Right Anti Join
        • Right Outer Join
      • Join Criteria
        • Matching Field Names
        • Condition
      • Configuring a Join Processor
    • JSON Parser
      • Schema
        • JSON Schema Format
        • DDL Schema Format
        • Schema Error Handling
      • Configuring a JSON Parser Processor
    • Pivot
      • Generated Records
      • Configuring a Pivot Processor
    • Profile
      • Profile Statistics
      • Output Records
      • Configuring a Profile Processor
    • PySpark
      • Prerequisites
      • Provisioned Databricks Cluster Requirements
      • Develop the PySpark Code
        • Input and Output Variables
        • Referencing Fields
        • Examples
      • Configuring a PySpark Processor
    • Rank
      • Rank Processing
      • Rank Functions
      • Partition and Order By Fields
      • Configuring a Rank Processor
    • Repartition
      • Repartition Use Cases
      • Repartition Methods
        • Repartition by Number
        • Coalesce by Number
        • Repartition by Field Hash
        • Repartition by Field Range
      • Configuring a Repartition Processor
    • Scala
      • Scala Code
        • Scala Requirements
      • Input and Output Variables
      • Performing Spark SQL Queries
      • Examples
      • Configuring a Scala Processor
    • Slowly Changing Dimension
      • Slowly Changing Dimension Pipeline
      • Pipeline Configuration
        • Table Dimension Pipeline
        • File Dimension Pipeline
        • Grouped File Dimension Prerequisite
      • Slowly Changing Dimension Type
      • Tracking Fields
        • Timestamp Basis Field
      • Change Processing
      • Processor Output
      • Examples of a Slowly Changing Dimension
      • Configuring a Slowly Changing Dimension Processor
    • Snowflake Lookup
      • Required Privileges and Custom Roles
      • Pushdown Optimization
      • Configuring a Snowflake Lookup Processor
    • Sort
      • Sort by Multiple Fields
      • Configuring a Sort Processor
    • Spark SQL Expression
      • Expressions
      • Configuring a Spark SQL Expression Processor
    • Spark SQL Query
      • Spark SQL Query Syntax
        • Referencing Fields
        • Examples
      • Configuring a Spark SQL Query Processor
    • Stream Selector
      • Default Stream
      • Conditions
      • Configuring a Stream Selector Processor
    • Surrogate Key Generator
      • Starting Key Value
      • Configuring a Surrogate Key Generator Processor
    • Type Converter
      • Field Type Conversion
      • Configuring a Type Converter Processor
    • Union
      • Operation Examples
      • Configuring a Union Processor
    • Window
      • Window Types
      • Sliding Window Condition
      • Configuring a Window Processor
    • XML Parser
      • Schema
        • JSON Schema Format
        • DDL Schema Format
        • Schema Error Handling
      • Configuring an XML Parser Processor
  • Destinations
    • Overview
    • ADLS Gen2
      • Prerequisites
        • Retrieve Authentication Information
      • Write Mode
        • Spark Requirement to Overwrite Related Partitions
      • Partition Files
      • Data Formats
      • Configuring an ADLS Gen2 Destination
    • Amazon Redshift
      • Installing a JDBC Driver
      • Credentials and Writing to Redshift
      • Server-Side Encryption
      • Write Mode
      • Partitioning
      • Configuring an Amazon Redshift Destination
    • Amazon S3
      • Prerequisites
      • URI Scheme
      • Authentication Method
      • Server-Side Encryption
      • Write Mode
        • Spark Requirement to Overwrite Related Partitions
      • Partition Objects
      • Data Formats
      • Configuring an Amazon S3 Destination
    • Azure Event Hubs
      • Prerequisites
      • Data Formats
      • Configuring an Azure Event Hubs Destination
    • Azure SQL
      • Partitioning
      • Write Mode
      • Configuring an Azure SQL Destination
    • Delta Lake
      • Storage Systems
      • ADLS Gen2 Prerequisites
        • Retrieve Authentication Information
      • Amazon S3 Credential Mode
      • Creating a Table or Managed Table
        • Creating a Delta Lake Table
        • Creating a Managed Delta Lake Table
      • Partitioning
      • Write Mode
        • Append Data
        • Overwrite Data
          • Overwrite Condition
        • Upsert Using Merge
          • Change Data Capture Example
        • Update Table
        • Delete from Table
      • Writing to a Local File System
      • Configuring a Delta Lake Destination
    • File
      • Directory Path
      • Write Mode
        • Spark Requirement to Overwrite Related Partitions
      • Partition Files
      • Data Formats
      • Configuring a File Destination
    • Google Big Query
      • Prerequisite
      • Write Mode
        • Merge Properties
      • Configuring a Google Big Query Destination
    • Hive
      • Partitioning
      • File Formats
      • Data Drift Handling
        • Column Order
      • Additional Hive Configuration Properties
      • Configuring a Hive Destination
    • JDBC
      • Database Vendors and Drivers
      • Installing the JDBC Driver
      • Write Mode
      • Partitioning
      • Configuring a JDBC Destination
    • Kafka
      • Generated Messages and Kafka Message Keys
      • Kafka Security
      • Data Formats
      • Configuring a Kafka Destination
    • Kudu
      • Configuring a Kudu Destination
    • Snowflake
      • Required Privileges and Custom Roles
      • Write Mode
        • Merge Properties
      • Snowflake Data Types
      • Configuring a Snowflake Destination
    • Unity Catalog
      • Table Creation
      • Partitioning
      • Write Mode
        • Upsert Using Merge
      • Configuring a Unity Catalog Destination
  • Preview
    • Overview
      • Preview Availability
      • Spark for Preview
      • Writing to Destinations
      • Preview Data Types
    • Preview Codes
    • Processor Output Order
    • Input and Output Schema for Stages
    • Previewing a Pipeline
    • Editing Properties
  • Troubleshooting
    • Pipelines
    • Origins
  • StreamSets Expression Language
    • StreamSets Expression Language
      • Basic Syntax
      • Expression Completion
        • Usage and Tips
      • Expression Examples
    • Functions
      • Base64 Functions
      • Batch Functions
      • Credential Functions
      • File Functions
      • Job Functions
      • Math Functions
      • Pipeline Functions
      • String Functions
      • Time Functions
      • Miscellaneous Functions
    • Constants
    • Datetime Variables
    • Literals
    • Operators
      • Operator Precedence
    • Reserved Words
• Connections
  • Connection Management
    • Overview
      • Connection Requirements
        • Data Collector Versions
        • Transformer versions
    • Getting Started with Connections
      • Step 1. Create a Connection
      • Share the Connection
      • Use the Connection
    • Expressions in Connection Properties
    • Securing Connections
      • Secure Connection Objects
      • Secure Connection Values
    • Connection Tags
    • Using Connections in Pipeline Fragments
    • Managing Connections
      • Filtering Connections
      • Testing Connections
      • Viewing Connection Usage
      • Editing Connections
      • Upgrading a Connection
      • Deleting Connections
  • Connection Types
    • Overview
    • Amazon
      • Amazon EMR Cluster Manager Connection
      • Amazon EMR Serverless Connection
      • Amazon Kinesis Firehose Connection
      • Amazon Kinesis Streams Connection
      • Amazon Redshift Connection
      • Amazon S3 Connection
      • Amazon SQS Connection
      • Amazon Security
        • Assume Another Role
          • Assume Role Methods
          • Create the Trust Policy
          • Configure Connections to Assume a Role
      • Amazon Connection Properties
        • Common Properties
        • Advanced Properties
        • Amazon Redshift Properties
        • EMR Cluster Manager Properties
        • EMR Serverless Properties
    • Azure Blob Storage
      • Prerequisites
      • Azure Blob Storage Connection Properties
    • Azure Data Lake Storage Gen2
      • Prerequisites
        • Retrieve Authentication Information
      • Azure Data Lake Storage Gen2 Connection Properties
    • Azure Synapse
      • Azure Synapse Authentication
      • Azure Synapse Connection Properties
    • Cassandra
      • Cassandra Connection Properties
    • CoAP Client
      • CoAP Client Connection Properties
    • Couchbase
      • Couchbase Connection Properties
    • Databricks
      • Databricks Connection Properties
    • Elasticsearch
      • Elasticsearch Security
      • Elasticsearch Connection Properties
    • Google Cloud
      • Google BigQuery Connection
      • Google Cloud Storage Connection
      • Google Pub/Sub Connection
      • Google Cloud Credentials
        • Default Credentials
        • Credentials in a File
        • Credentials in a Connection Property
      • Google Cloud Connection Properties
    • Hive
      • Hive Connection Properties
    • IBM Cloud Object Storage
      • IBM Cloud Object Storage properties
    • IBM Db2
      • IBM Db2 Properties
    • IBM watsonx.data
      • IBM watsonx.data properties
    • InfluxDB
      • InfluxDB Connection Properties
    • InfluxDB 2.x
      • InfluxDB 2.x Connection Properties
    • JDBC
      • JDBC Connection Properties
    • JMS
      • JMS Connection Properties
    • Kafka
      • Kafka Connection Properties
      • Kafka Security
        • Prerequisite Tasks
        • SASL Authentication Credentials
          • Providing PLAIN Credentials
          • Providing Kerberos Credentials
            • Using a Credential Store
        • Enabling SSL/TLS Encryption
        • Enabling SSL/TLS Encryption and Authentication
        • Enabling SASL Authentication
        • Enabling SASL Authentication on SSL/TLS
        • Enabling Confluent Cloud authentication
        • Enabling Custom Authentication
    • Kudu
      • Kudu Connection Properties
    • MongoDB Atlas
      • Credentials
      • Enabling SSL/TLS
      • MongoDB Atlas Connection Properties
    • MQTT
      • MQTT Connection Properties
    • MySQL
      • MySQL Connection Properties
    • OPC UA Client
      • OPC UA Client Connection Properties
    • Oracle
      • Oracle Connection Properties
    • Orchestrator
      • Orchestrator Connection Properties
    • PostgreSQL
      • PostgreSQL Connection Properties
    • RabbitMQ
      • RabbitMQ Connection Properties
    • Redis
      • Redis Connection Properties
    • Salesforce
      • Authentication Types
      • Changing the API Version
      • Salesforce Connection Properties
    • SFTP/FTP/FTPS
      • Credentials
      • SFTP/FTP/FTPS Connection Properties
    • Snowflake
      • Snowflake Connection
        • Snowflake Roles and Privileges
        • Using Custom Roles
        • Snowflake Connection Properties
      • Snowpipe Connection
        • Snowpipe Connection Properties
    • Splunk
      • Splunk Connection Properties
    • SQL Server
      • SQL Server Connection Properties
    • Teradata
      • Teradata Connection Properties
    • Web Client
      • Web Client Connection Properties
• Jobs
  • Jobs Overview
  • Number of Pipeline Instances
    • Scaling Out Active Jobs
    • Rules and Guidelines
  • Data Collector Pipeline Failover
    • Failover and Number of Pipeline Instances
    • Failover Retries
    • Determining When to Enable Failover
    • Configure Origin Systems for Failover
    • Enabling Pipeline Failover
    • Balancing Jobs Enabled for Failover
      • Balance Specific Jobs
      • Balance Jobs on Specific Data Collectors
      • Comparing Balance Jobs and Synchronize Jobs
  • Transformer Pipeline Failover
    • Failover and Backup Transformers
    • Failover Requirements
    • Failover Retries
    • Enabling Pipeline Failover
  • Runtime Parameters
    • Using Functions for Parameter Values
  • Job Templates
    • Job Template Creation
    • Job Instance Creation
    • Suffix for job instance names
    • Parameter Values for Job Instances
  • Job Tags
  • Managing Jobs
    • Creating Jobs and Job Templates
    • Starting Jobs and Job Templates
      • Starting Jobs
      • Starting Job Templates
    • Filtering Jobs and Job Templates
    • Synchronizing Jobs
    • Job Offsets
      • Origins that Maintain Offsets
      • Resetting the Origin for Jobs
      • Uploading an Initial Offset File
    • Editing the Latest Pipeline Version
    • Upgrading to the Latest Pipeline Version
    • Stopping Jobs
      • Forcing a Job to Stop
    • Scheduling Jobs
    • Editing Jobs
    • Duplicating Jobs
    • Deleting Jobs
  • Monitoring Jobs
    • Job Status
      • Pipeline Status
      • Pipeline Status Examples
    • Jobs and Unresponsive Data Collector Engines
      • Engine Gracefully Shuts Down
      • Engine Loses Connection
      • Engine Unexpectedly Shuts Down
    • Acknowledging Job Errors
    • Resetting Metrics for Jobs
    • Monitoring Stage-Related Errors
    • Logs
      • Viewing the Execution Engine Log
      • Viewing the Spark Driver Log
    • Cluster and Spark URLs
    • Snapshots
      • Failure Snapshots
        • Viewing a Failure Snapshot
      • Capturing and Viewing a Snapshot
      • Renaming a Snapshot
      • Downloading a Snapshot
      • Deleting a Snapshot
    • Viewing the Job Run History
      • Viewing a Run Summary
  • Troubleshooting
• Scheduler
  • Scheduler Overview
  • Scheduled Task Types
    • Jobs
      • Batch and Streaming Jobs
  • Time Zone
  • Cron Expression
  • Start and End Times
  • Missed Execution Handling
  • Creating Scheduled Tasks
  • Monitoring Scheduled Tasks
    • Scheduled Task Status
    • Viewing Runs
    • Viewing the Audit
  • Managing Scheduled Tasks
• Topologies
  • Topologies Overview
  • Map Jobs in a Topology
    • Connect Multiple Jobs to a Single System
    • Create Topologies
      • Creating an Empty Topology
      • Creating a Topology for Multiple Jobs
    • Mapping Jobs and Systems in a Topology
    • Auto Discover Connecting Systems
    • Managing Jobs from a Topology
    • Customizing System Icons
  • Measure Topology Performance
    • Monitor Jobs and Systems
  • Monitor Topologies with Data SLAs
    • Creating Data SLAs
    • Managing Data SLA Alerts
  • Default Topology
  • Auto Fix an Invalid Topology
  • Topology History
  • Renaming a Topology
  • Duplicating a Topology
  • Deleting Topology Versions
• Reports
  • Reports View
• Subscriptions
  • Subscriptions Overview
  • Enabling Events to Trigger Subscriptions
  • Events
    • Event Conditions
    • Advanced Expressions
      • Expression Completion
      • Example Expressions
  • Actions
    • Webhook Action
      • Payload
    • Email Action
  • Parameters
    • Using Parameters in Actions
    • Using Parameters in Event Conditions
    • General Event Parameters
    • Data SLA Triggered Parameters
    • Execution Engine Not Responding Parameters
    • Job Status Change Parameters
    • Max Global Failover Retries Exhausted Parameters
    • Pipeline Committed Parameters
    • Pipeline Status Change Parameters
    • Report Generated Parameters
  • Examples
    • Send a Message to a Slack Channel
    • Call an External API
    • Sending an Email
  • Creating Subscriptions
  • Troubleshooting
• Data Collectors
  • Data Collectors Overview
    • Example
    • Data Collector Versions
  • Manually Administered Data Collectors
    • Register Data Collectors
      • Registering from Control Hub
      • Registering from the Command Line Interface
      • Registering from Cloudera Manager
    • Using a Publicly Accessible URL
    • Deleting Unregistered Tokens
    • Regenerate a Token
      • Regenerating from Control Hub
      • Regenerating from Cloudera Manager
    • Unregister Data Collectors
      • Unregistering from Control Hub
      • Unregistering from the Command Line Interface
      • Unregistering from Control Hub and Cloudera Manager
  • Authoring Data Collectors
    • System Data Collector
    • Registered Data Collector
  • Labels
  • Resource Thresholds
    • Defining Resource Thresholds
  • Monitoring Data Collectors
    • Performance
    • Configuration Properties
    • Metrics
    • Thread Dump
    • Support Bundles
    • Viewing Logs
    • Directories
    • Health Inspector
    • Pipeline Status
  • Trusted Domains for Data Collectors
  • Using an HTTP or HTTPS Proxy Server
  • Deactivating a Registered Data Collector
  • Control Hub Configuration File
  • Troubleshooting
• Transformers
  • Installation
    • Overview
    • Installation Requirements
      • System Requirements
      • Cluster Compatibility Matrix
      • Scala, Spark, and Java JDK Requirements
      • Spark Versions and Available Features
        • Spark Shuffle Service Requirement
      • Configure the Open File Limit
      • Default Port
      • Docker Image Requirement
      • Hadoop YARN Requirements
        • Directories
        • JDBC Driver
        • Memory
    • Installing Transformer
      • Choosing an Installation Package
      • Installing when Spark Runs Locally
      • Installing when Spark Runs on a Cluster
        • Enabling Kerberos for Hadoop YARN Clusters
          • Enabling Proxy Users
          • Enabling the Properties File as the Keytab Source
      • Installation through the Azure Marketplace
        • Installing Transformer on Azure
        • Installing StreamSets for Databricks on Azure
      • Configuring Directories (Tarball Installation)
      • Run Transformer from Docker
    • Understanding the Spark Cluster Callback URL
      • Cluster Callback URL Properties
    • Accessing Log File Information
      • Transformer Log Format
      • Viewing the Transformer Log
      • Modifying the Transformer Log Level
    • Protecting Sensitive Data in Configuration Files
    • Customization with Environment Variables
      • Modifying Environment Variables
      • Transformer Directories
      • User and Group for Service Start
      • Java Configuration Options
        • Java Heap Size
        • Remote Debugging
        • Garbage Collector
        • Heap Dump Creation
      • Security Manager
        • Protected Directories
      • Root Classloader
    • Enabling HTTPS
      • Prerequisites
      • Create a Keystore File
      • Create a Truststore File
      • Configure Transformer to Use HTTPS
    • Using a Reverse Proxy
    • Credential Stores
      • Enabling Credential Stores
      • Group Access to Secrets
      • AWS Secrets Manager
        • Step 1. Configure the Credential Store Properties
        • Step 2. Call Secrets from the Pipeline
      • Azure Key Vault
        • Prerequisites
        • Step 1. Configure the Credential Store Properties
        • Step 2. Call Secrets from the Pipeline
      • CyberArk
        • Step 1. Configure the Credential Store Properties
        • Step 2. Call Secrets from the Pipeline
      • Google Secret Manager
        • Authentication
        • Step 1. Configure the Credential Store Properties
        • Step 2. Call Secrets from the Pipeline
      • Hashicorp Vault
        • Step 1. Configure the Credential Store Properties
        • Step 2. Call Secrets from the Pipeline
      • Java Keystore
        • Step 1. Configure Credential Store Properties
        • Step 2. Add Secrets to the Java Keystore
        • Step 3. Call Secrets from the Pipeline
        • jks-credentialstore Command
    • Stage-Related Prerequisites
      • Local Pipeline Prerequisites for Amazon S3 and ADLS Gen2
        • Ensure Access to Hadoop Client Libraries
        • Use the Transformer Stage Library
      • PySpark Processor Prerequisites
        • Existing Databricks Cluster
        • Existing EMR Cluster
        • Other Existing Clusters and Local Pipelines
      • Scala Processor and Preprocessing Script Prerequisites
        • Set the SPARK_HOME Environment Variable
    • External Libraries
      • Setting Up an External Directory
      • Installing from Package Manager
      • Installing from Stage Properties
      • Installing Manually
        • Tarball and RPM Installations
      • Managing External Libraries
    • Starting Transformer
      • Starting Transformer Manually
      • Starting Transformer as a Service
    • Uninstallation
      • Uninstalling the Tarball
      • Uninstalling the RPM Package
  • Upgrade
    • Overview
    • Pre Upgrade Tasks
      • Verify Installation Requirements
    • Upgrade an Installation from the Tarball
      • Step 1. Shut Down the Previous Version
      • Step 2. Back Up the Previous Version
      • Step 3. Install the New Version
      • Step 4. Update Environment Variables
      • Step 5. Update the Configuration Files
      • Step 6. Start the New Version of Transformer
    • Upgrade an Installation from the RPM Package
      • Step 1. Shut Down the Previous Version
      • Step 2. Back Up the Previous Version
      • Step 3. Install the New Version
      • Step 4. Update Environment Variables
      • Step 5. Update the Configuration Files
      • Step 6. Start the New Version of Transformer
    • Post Upgrade Tasks
      • Update Control Hub On-Premises
      • Review Dockerfiles for Custom Docker Images
      • Review Firewall Rules and Automation for Docker Images
      • Review Amazon Stages that Assume Roles
      • Upgrade Clusters and Transformer to Spark 3.3 or Later
      • Review Pipelines with Removed Functionality
      • Review Dockerfiles for Custom Docker Images
      • Review Delta Lake Stages Using Transformer-Provided Libraries
      • Review Drivers for Azure SQL Destinations
      • Review Lookup Processors
      • Review Dockerfiles for Custom Docker Images
      • Manage Underscores in Snowflake Connection Information
      • Update SDK Code for Database-Vendor-Specific JDBC Origins
      • Review Use of Generated XML Files
      • Install Java JDK 11 for Scala 2.12 Installations
      • Access Databricks Job Details
      • Update ADLS Stages in HDInsight Pipelines
      • Update Keystore and Truststore Location
      • Update Kubernetes Pipelines
      • Update Drivers on Older Hadoop Clusters
      • Enable the Spark External Shuffle Service
      • Update Spark SQL Expression Pipelines
    • Managing Cluster Version Changes
    • Troubleshooting an Upgrade
  • Transformer Registration
    • Transformer Registration Overview
      • Transformer Versions
    • Register Transformer
      • Registering from Control Hub
      • Registering from the Command Line Interface
      • Using a Publicly Accessible URL
      • Using a Proxy Server
      • Deleting Unregistered Tokens
      • Deactivating a Registered Transformer
      • Regenerating a Token
    • Unregister Transformer
      • Unregistering from Control Hub
      • Unregistering from the Command Line Interface
    • Labels
    • Disconnected Mode
    • Resource Thresholds
      • Defining Resource Thresholds
    • Monitoring Transformers
      • Performance
      • Configuration Properties
      • Metrics
      • Thread Dump
      • Support Bundles
      • Viewing Logs
      • Directories
      • Pipeline Status
    • Control Hub Configuration File
• Provisioned Data Collectors
  • Provisioned Data Collectors Overview
  • Provisioning Requirements
    • StreamSets Control Agent Versions
      • Version 5.3.2
      • Version 5.3.1
      • Version 5.3.0
      • Version 5.2.1
      • Version 5.2.0
      • Version 5.1.0
      • Version 5.0.0
  • Provision Data Collectors
    • Step 1. Create a Custom Image
    • Step 2. Create a Provisioning Agent
    • Step 3. Define a Deployment YAML Specification
      • Deployment Sample
      • Deployment and Horizontal Pod Autoscaler Sample
      • Attributes for AWS Fargate with EKS
    • Step 4. Create a Deployment
    • Step 5. Start the Deployment
  • Labels
  • Managing Provisioning Agents
    • Applying Changes to Provisioning Agents
  • Managing Deployments
    • Manually Scale the Deployment
    • Deployment Status
• Labels
  • Labels Overview
    • Labels and Execution Component
    • Labels and Pipeline Type
    • Label Examples
  • Default Label
  • Working with Multiple Labels
  • Using Labels to Support Pipeline Failover
  • Assigning Labels to a Data Collector or Transformer
    • Using the Control Hub Configuration File
    • Using the Control Hub UI
  • Assigning Labels to a Deployment
• Export and Import
  • Export and Import Overview
    • Object ID on Export
    • Permissions on Export
  • Pipeline, Fragment, and Topology Export and Import
    • Import Objects from a View
    • Import a Pipeline Version from the Pipeline Canvas
    • Parent Versions
  • Job Export and Import
  • Objects that Use Connections
  • Exporting Objects
    • Exporting Pipelines
    • Exporting Fragments
    • Exporting Jobs
    • Exporting Topologies
  • Importing Objects
    • Importing Pipelines
    • Importing a Pipeline Version from the Canvas
    • Importing Fragments
    • Importing Jobs
    • Importing Topologies
• Organization Security
  • Organization Security Overview
  • Authentication
    • Control Hub Authentication
    • SAML Authentication
      • Signed Messages and Assertions
      • Tasks that Require Control Hub Credentials
      • Enabling SAML Authentication
      • Step 1. Register Control Hub as a Service Provider
      • Step 2. Configure SAML Authentication for the Organization
      • Step 3. Map Control Hub Users to Identity Provider Users
      • Troubleshooting
  • Users and Groups
    • Creating User Accounts
    • Creating Groups
    • Adding Users to Groups
    • Activating or Deactivating Users
    • Resetting a Password
  • Roles
    • Role Descriptions
    • Common Role Assignments
  • Permissions
    • Enabling Permission Enforcement
    • Connection Permissions
    • Data SLA Permissions
    • Deployment Permissions
    • Engine Permissions
    • Job Permissions
    • Pipeline Fragment Permissions
    • Pipeline Permissions
    • Provisioning Agent Permissions
    • Report Task Permissions
    • Scheduled Task Permissions
    • Subscription Permissions
    • Topology Permissions
    • Sharing Objects
    • Changing the Object Owner
  • Requirements for Common Tasks
    • Connection Tasks
    • Data SLA Tasks
    • Engine Tasks
    • Job Tasks
    • Pipeline Fragment Tasks
    • Pipeline Tasks
    • Provisioning Tasks
    • Report Tasks
    • Scheduler Tasks
    • Subscription Tasks
    • Topology Tasks
  • Organization Administration
    • Organization Configuration
      • Organization Details
      • Organization Configuration Window
    • Active User Sessions
    • Audits
  • Rate Limiting
• Glossary
  • Glossary of Terms
• StreamSets Legacy SDK