Google Pub/Sub Publisher
The Google Pub/Sub Publisher destination publishes messages to a Google Pub/Sub topic. For information about supported versions, see Supported Systems and Versions in the Data Collector documentation.
When you configure the destination, you define the Google Pub/Sub topic ID to write messages to. You also define the project ID and credentials to use when connecting to Google Pub/Sub.
By default, the Google Pub/Sub Publisher destination writes messages in batches. Using advanced properties, you can configure the conditions that trigger writing a new batch or disable batch processing to write messages individually. You can also configure the action the destination takes when it reads messages faster than it can write messages.
A Google Pub/Sub message contains a payload and optional user-defined attributes that describe the payload content. When records include record header attributes, the Google Pub/Sub Publisher destination includes the record header attributes in message attributes. The destination does not include internal record header attributes in message attributes. For more information about record header attributes, see Record Header Attributes.
You can also use a connection to configure the destination.
Credentials
When the Google Pub/Sub Publisher destination publishes messages to a Google Pub/Sub topic, the destination must pass credentials to Google Pub/Sub.
- Google Cloud default credentials
- Credentials in a file
- Credentials in a stage property
For details on how to configure each option, see Security in Google Cloud Stages.
Data Formats
Google Pub/Sub Publisher writes data to Google Pub/Sub based on the data format that you select. You can use the following data formats:
- Avro
- The stage writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
- XML
- The destination creates a valid XML document for each record. The
destination requires the record to have a single root field that
contains the rest of the record data. For details and
suggestions for how to accomplish this, see Record Structure Requirement.
The destination can include indentation to produce human-readable documents. It can also validate that the generated XML conforms to the specified schema definition. Records with invalid schemas are handled based on the error handling configured for the destination.
Configuring a Google Pub/Sub Publisher Destination
Configure a Google Pub/Sub Publisher destination to write messages to a Google Pub/Sub topic.