MapR Streams Producer
The MapR Streams Producer destination writes messages to MapR Streams.
MapR is now HPE Ezmeral Data Fabric. At times, this documentation uses "MapR" to refer to both MapR and HPE Ezmeral Data Fabric. For information about supported versions, see Supported Systems and Versions.
When you configure a MapR Streams Producer, you configure the topic, partition strategy, and other general properties. You configure the data type and related properties, and you can optionally add additional MapR Streams properties and supported Kafka properties.
Before you use any MapR stage in a pipeline, you must perform additional steps to enable Data Collector to process MapR data. For more information, see MapR Prerequisites.
Data Formats
- Avro
- The stage writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
Runtime Topic Resolution
MapR Streams Producer can write a record to the topic based on an expression. When MapR Streams Producer evaluates a record, it calculates the expression based on record values and writes the record to the resulting topic.
When performing runtime topic resolution, MapR Streams Producer can write to any topic by default. You can create an allowlist of topics to limit the number of topics the destination attempts to use. When you create an allowlist, any record that resolves to an unlisted topic is sent to the stage for error handling. Use an allowlist when record data might resolve to invalid topic names.
Partition Strategy
The partition strategy determines how to write data to partitions. You can use a partition strategy to balance the work load or to write data semantically.
- Round-Robin
- Writes each record to a different partition using a cyclical order. Use for load balancing.
- Random
- Writes each record to a different partition using a random order. Use for load balancing.
- Expression
- Writes each record to a partition based on the results of the partition expression. Use to perform semantic partitioning.
- Default
- Writes each record using the default partition strategy that MapR Streams provides.
Additional Properties
You can add custom configuration properties to MapR Streams Producer.
You can add any valid configuration property. When you add a property, enter the exact property name and the value. MapR Streams Producer does not validate the property names or values.
You can use any MapR or Kafka property supported by MapR Streams. For more information, see the MapR documentation.
If custom configurations conflict with other stage properties, the stage generates an error unless you select the Override Stage Configurations check box. With the check box selected, the custom configurations override other stage properties. For information about the necessary properties, see the MapR documentation.
Configuring a MapR Streams Producer Destination
The MapR Streams Producer destination writes messages to MapR Streams.