HTTP Client
The HTTP Client destination writes data to an HTTP endpoint. The destination sends requests to an HTTP resource URL. For information about supported versions, see Supported Systems and Versions.
The newer Web Client destination provides much of the same functionality as the HTTP Client destination. It also provides functionality not available in the HTTP Client destination. For more information, see Comparing Web Client and HTTP Client Destinations.
Use the HTTP Client destination to perform a range of standard requests or use an expression to determine the request for each record. The destination can also send responses to a microservice origin when used in a microservice pipeline.
When you configure the HTTP Client destination, you define the resource URL, optional headers, and method to use for the requests. You configure the destination to generate one request for each record or to generate a single request containing all records in the batch.
You can configure the timeout, request transfer encoding, and authentication type. You can configure the destination to use the Gzip or Snappy compression format to write the messages. You can also configure the processor to log request and response information.
You can optionally use an HTTP proxy and configure SSL/TLS properties. You can also configure the destination to use the OAuth 2 protocol to connect to an HTTP service.
When you want the destination to send responses to a microservice origin within a microservice pipeline, you specify the type of response to send.
HTTP Method
- GET
- PUT
- POST
- DELETE
- HEAD
- PATCH
- Expression - An expression that evaluates to one of the other methods.
Expression Method
The Expression method allows you to write an expression that evaluates to a standard HTTP method. Use the Expression method to generate a workflow. For example, you can use an expression that passes data to the server (PUT) based on the data in a field.
Headers
- Headers
- Additional Security Headers
You can define headers in either property. However, only additional security headers support using credential functions to retrieve sensitive information from supported credential stores.
If you define the same header in both properties, additional security headers take precedence.
Number of Requests
The HTTP Client destination can generate one HTTP request for each record, or it can generate a single request containing all records in the batch.
Configure the destination to generate requests in one of the following ways:
- Multiple requests per batch
- By default, the destination generates one HTTP request for each record in the batch and sends multiple requests at a time. To preserve record order, the destination waits until all requests for the entire batch are completed before processing the next batch.
- Single request per batch
- If you enable the One Request per Batch property, the destination generates a single HTTP request containing all records in the batch.
Logging Request and Response Data
The HTTP Client destination can log request and response data to the Data Collector log.
When enabling logging, you configure the following properties:
- Verbosity
-
The type of data to include in logged messages:
- Headers_Only - Includes request and response headers.
- Payload_Text - Includes request and response headers as well as any text payloads.
- Payload_Any - Includes request and response headers and the payload, regardless of type.
- Log Level
- The level of messages to include in the Data Collector log. When you select a level, higher level messages are also logged. That is, if you select the Warning log level, then Severe and Warning messages are written to the Data Collector log.
- Max entity size
-
The maximum size of message data to write to the log. Use to limit the volume of data written to the Data Collector log for any single message.
Send Microservice Responses
The HTTP Client destination can send responses to a microservice origin when you use the destination in a microservice pipeline.
- All successfully written records.
- Responses from the destination system - For information about the possible responses, see the documentation for the destination system.
OAuth 2 Authorization
The HTTP Client destination can use the OAuth 2 protocol to connect to an HTTP service.
The HTTP Client destination can use the OAuth 2 protocol to connect to an HTTP service that uses basic, digest, or universal authentication, OAuth 2 client credentials, OAuth 2 username and password, or OAuth 2 JSON Web Tokens (JWT).
The OAuth 2 protocol authorizes third-party access to HTTP service resources without sharing credentials. The HTTP Client destination uses credentials to request an access token from the service. The service returns the token to the destination, and then the destination includes the token in a header in each request to the resource URL.
- Client credentials grant
-
HTTP Client sends its own credentials - the client ID and client secret or the basic, digest, or universal authentication credentials - to the HTTP service. For example, use the client credentials grant to process data from the Twitter API or from the Microsoft Azure Active Directory (Azure AD) API.
For more information about the client credentials grant, see https://tools.ietf.org/html/rfc6749#section-4.4.
- Resource owner password credentials grant
-
HTTP Client sends the credentials for the resource owner - the resource owner username and password - to the HTTP service. Or, you can use this grant type to migrate existing clients using basic, digest, or universal authentication to OAuth 2 by converting the stored credentials to an access token.
For example, use this grant to process data from the Getty Images API. For more information about using OAuth 2 to connect to the Getty Images API, see http://developers.gettyimages.com/api/docs/v3/oauth2.html.
For more information about the resource owner password credentials grant, see https://tools.ietf.org/html/rfc6749#section-4.3.
- JSON Web Tokens
-
HTTP Client sends a JSON Web Token (JWT) to an authorization service and obtains an access token for the HTTP service. For example, use JSON Web Tokens to process data with the Google API.
Let’s look at some examples of how to configure authentication and OAuth 2 authorization to process data from Twitter, Microsoft Azure AD, and Google APIs.
Example for Twitter
To use OAuth 2 authorization to write to Twitter, configure HTTP Client to use basic authentication and the client credentials grant.
For more information about configuring OAuth 2 authorization for Twitter, see https://developer.twitter.com/en/docs/authentication/oauth-2-0/application-only.
Example for Microsoft Azure AD
To use OAuth 2 authorization to write to Microsoft Azure AD, configure HTTP Client to use no authentication and the client credentials grant.
For more information about configuring OAuth 2 authorization for Microsoft Azure AD, see https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-protocols-oauth-code.
Example for Google
Configure the HTTP Client destination to use OAuth 2 authorization to write to Google service accounts. The stage sends a JSON Web Token in a request to the Google Authorization Server and obtains an access token for calls to the Google API.
Before you configure the stage, create a service account and delegate domain-wide authority to the service account. For details, see the Google Identity documentation: Using OAuth 2.0 for Server to Server Applications.
For more information about Google service accounts, see the Google Cloud documentation: Understanding service accounts.
For more information about configuring OAuth 2 authorization for Google, see the Google Identity documentation: Using OAuth 2.0 to Access Google APIs.
Data Formats
HTTP Client writes data to HTTP endpoints based on the data format that you select.
The HTTP Client destination processes data formats as follows:
- Avro
- The stage writes records based on the Avro schema. You can use one of the following methods to specify the location of the Avro schema definition:
- Binary
- The stage writes binary data to a single field in the record.
- Delimited
- The destination writes records as delimited data. When you use this data format, the root field must be list or list-map.
- JSON
- The destination writes records as JSON data. You can use one of
the following formats:
- Array - Each file includes a single array. In the array, each element is a JSON representation of each record.
- Multiple objects - Each file includes multiple JSON objects. Each object is a JSON representation of a record.
- Protobuf
- Writes one record in a message. Uses the user-defined message type and the definition of the message type in the descriptor file to generate the message.
- SDC Record
- The destination writes records in the SDC Record data format.
- Text
- The destination writes data from a single text field to the destination system. When you configure the stage, you select the field to use.
Configuring an HTTP Client Destination
Configure an HTTP Client destination to write data to an HTTP endpoint.