Control Hub API
The Control Hub API processor sends requests to the Control Hub REST API and writes data from the response to a specified output field.
The Control Hub API processor is an orchestration stage that you use in orchestration pipelines. Orchestration stages perform tasks, such as schedule and start pipelines and Control Hub jobs, that you can use to create an orchestrated workflow across IBM StreamSets. For example, an orchestration pipeline can use the Cron Scheduler origin to generate a record every weekday at 5 PM and trigger the Control Hub API processor, which calls a Control Hub REST API to stop a Control Hub job that runs during business hours.
When you configure the Control Hub API processor, you specify the URL for the API, the field to write the response to, any headers to include with the request, and the method to use. The method can specify a standard HTTP request or an expression that determines the request for each record. For some methods, you can specify data to include with the request.
You can configure the processor to log request and response information.
You can also configure the timeout and maximum number of parallel requests. You can optionally use an HTTP proxy and configure SSL/TLS properties.
You can also use a connection to configure the processor.
Finding the URL to a Control Hub REST API
Each Control Hub REST API has a unique URL. The URL contains the Control Hub URL and the path to the API.
You can find the URL in Control Hub.
HTTP Method
- GET
- PUT
- POST
- DELETE
- Expression - An expression that evaluates to one of the other methods.
Specify a method valid for the API call.
You can use the Expression method to set the HTTP method based on the data in a field.
For example, the same API call,
/pipelinestore/rest/v1/pipelineCommit/{commitId}
, can either
retrieve, save, or delete a pipeline commit, depending on the specified method.
Therefore, you might configure the Control Hub API processor to use that API call with
an expression that returns either the GET, POST, or DELETE method based on data in a
field.
Parallel Requests
The Control Hub API processor can send multiple requests simultaneously.
To preserve record order, the processor waits until all the requests from an entire batch are completed before sending requests from the next batch.
You can specify the maximum number of parallel requests. Default is 1. Increasing the number of parallel requests can improve performance but increases the load on the Control Hub server. Network latency can also significantly impact the performance of this processor.
Logging Request and Response Data
The Control Hub API processor can log request and response data to the Data Collector log.
When enabling logging, you configure the following properties:
- Verbosity
-
The type of data to include in logged messages:
- Headers_Only - Includes request and response headers.
- Payload_Text - Includes request and response headers as well as any text payloads.
- Payload_Any - Includes request and response headers and the payload, regardless of type.
- Log Level
- The level of messages to include in the Data Collector log. When you select a level, higher level messages are also logged. That is, if you select the Warning log level, then Severe and Warning messages are written to the Data Collector log.
- Max entity size
-
The maximum size of message data to write to the log. Use to limit the volume of data written to the Data Collector log for any single message.
Generated Record
The Control Hub API processor updates the orchestration record that it receives with the response from the Control Hub REST API. The response is placed in the output field specified in the stage properties. The fields included in the response depends on the API that the Control Hub API processor calls.
For example, the following preview shows the currentStatus
field that
was added to the orchestration record by the Control Hub API processor. The field
contains the response from a currentStatus
API call. The call requests
the status of the job that completed earlier in the orchestration pipeline and returns
additional information about the job:
Configuring a Control Hub API Processor
Configure a Control Hub API processor to call a Control Hub REST API. The Control Hub API processor is an orchestration stage that you use in orchestration pipelines.