Azure Event Hub Producer

The Azure Event Hub Producer writes data to Microsoft Azure Event Hub.

To write to Microsoft Azure Data Lake Storage, use the Azure Data Lake Storage target. To write to Microsoft Azure IoT Hub, use the Azure IoT Hub Producer target. For information about supported versions, see Supported systems and versions.

When you configure the Azure Event Hub Producer, you specify the Microsoft Azure event hub names and the connection string key to use.

Data formats

The Azure Event Hub Producer target writes data to Microsoft Azure Event Hub based on the data format that you select. You can use the following data formats:
Binary
The stage writes binary data to a single field in the record.
JSON
The target writes records as JSON data. You can use one of the following formats:
  • Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
  • Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
SDC Record
The target writes records in the SDC Record data format.
Text
The target writes data from a single text field to the target system. When you configure the stage, you select the field to use.
You can configure the characters to use as record separators. By default, the target uses a UNIX-style line ending (\n) to separate records.
When a record does not contain the selected text field, the target can report the missing field as an error or to ignore the missing field. By default, the target reports an error.
When configured to ignore a missing text field, the target can discard the record or write the record separator characters to create an empty line for the record. By default, the target discards the record.
XML
The target creates a valid XML document for each record. The target requires the record to have a single root field that contains the rest of the record data. For details and suggestions for how to accomplish this, see Record structure requirement.

The target can include indentation to produce human-readable documents. It can also validate that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling that is configured for the target.

Configuring an Azure Event Hub Producer target

About this task

Configure an Azure Event Hub Producer target to write data to Microsoft Azure Event Hub.

Procedure

  1. In the Properties panel, on the General tab, configure the following properties:
    General Property Description
    Name Stage name.
    Description Optional description.
    Stage Library Library version that you want to use.
    Produce Events Generates event records when events occur. Use for event handling.
    Required Fields Fields that must include data for the record to be passed into the stage.
    Tip: You might include fields that the stage uses.

    Records that do not include all required fields are processed based on the error handling that is configured for the flow.
    Preconditions Conditions that must evaluate to TRUE to allow a record to enter the stage for processing. Click Add to create more preconditions.

    Records that do not meet all preconditions are processed based on the error handling that is configured for the stage.
    On Record Error Error record handling for the stage:
    • Discard - Discards the record.
    • Send to Error - Sends the record to the flow for error handling.
    • Stop Flow - Stops the flow.
  2. On the Event Hub tab, configure the following properties:
    Event Hub Property Description
    Event Hub Name The event hub name.
    Connection String Key Connection string key to use to connect to Azure. The specified connection string key provides the shared access policy and namespace that is used to connect to Azure.

    To retrieve a connection string key, after you access the list of available shared access policies, click the policy name, and then copy the Primary Connection String value.

    The value typically begins with Endpoint.
    For more information about Microsoft Azure Event Hub, see the Event Hub documentation.
  3. On the Data Format tab, configure the following property:
    Data Format Property Description
    Data Format Format of data to be written. Use one of the following options:
  4. For binary data, on the Data Format tab, configure the following property:
    Binary Property Description
    Binary Field Path Field that contains the binary data.
  5. For JSON data, on the Data Format tab, configure the following properties:
    JSON Property Description
    JSON Content Method to write JSON data:
    • JSON Array of Objects - Each file includes a single array. In the array, each element is a JSON representation of each record.
    • Multiple JSON Objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
    Charset Character set to use to write data.
  6. For text data, on the Data Format tab, configure the following properties:
    Text Property Description
    Text Field Path Field that contains the text data to be written. All data must be incorporated into the specified field.
    Record Separator Characters to use to separate records. Use any valid Java string literal. For example, when you write to Windows, you might use \r\n to separate records.

    By default, the target uses \n.
    On Missing Field When a record does not include the text field, determines whether the target reports the missing field as an error or ignores the missing field.
    Insert Record Separator if No Text When configured to ignore a missing text field, inserts the configured record separator string to create an empty line.

    When not selected, discards records without the text field.
    Charset Character set to use to write data.
  7. For XML data, on the Data Format tab, configure the following properties:
    XML Property Description
    Pretty Format Adds indentation to make the resulting XML document easier to read, which also increases the record size.
    Validate Schema Validates that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling that is configured for the target.
    Important: Regardless of whether you validate the XML schema, the target requires the record in a specific format. For more information, see Record structure requirement.
    XML Schema The XML schema to use to validate records.