Microservice flows

Microservice flows overview

A microservice flow is a flow that creates a fine-grained service to perform a specific task.

For example, you might use a microservice flow to validate changes to user accounts. You can have the flow check for matching accounts to validate account creation requests, account updates, and account deletions. The flow can tag requests as actionable or not and send all responses back to the originating system for next steps.

You can create REST-based and message-based microservice flows that use standalone execution mode. Other microservice types and platforms are planned for future releases.

A microservice flow uses a microservice source to listen for or read JSON-formatted requests. In the flow, you can use any available processor to transform and route the records as needed. You can also use targets and executors to write data and execute additional actions as needed.

Then, use one or more microservice targets to send responses back to the source and write the records to a target system if necessary. The source then transmits JSON-formatted responses back to the source system.

For example, you might use the WebSocket Server source to listen at a WebSocket endpoint and process all authorized requests. In the flow, you use stages to process the data and route records to different targets based on whether the record is considered a success record or bad request record. You use two Send Response to Source targets to return different responses, 200 for OK and 400 for Bad Request. The source then passes the records with the responses back to the WebSocket endpoint. Or, you might use a different microservice target to write records to a target system while sending responses back to the source from the target system.

Stages for microservice flows

When you configure a microservice flow, you use a microservice source, any available Data Collector processor, target, and executor, and one or more microservice target.

Microservice sources

Microservice sources listen for or read JSON-formatted requests. They also send responses from microservice targets back to the source system.

You can use the following sources in microservice flows:
  • REST Service source - Listens on an HTTP endpoint, parses the contents of all authorized requests, and sends responses back to the originating REST API. Creates multiple threads to enable parallel processing in a multithreaded flow.
  • WebSocket Client source - Reads data from a WebSocket server endpoint. Can send responses back to the source system as part of a microservice flow.
  • WebSocket Server source - Listens on a WebSocket endpoint and processes the contents of all authorized WebSocket client requests. Creates multiple threads to enable parallel processing in a multithreaded flow. Can send responses back to the source system as part of a microservice flow.

Microservice targets

Most microservice targets perform two tasks. In addition to writing data to the target system, they send responses to the source stage, to be passed to the source system. When you configure microservice targets, you specify whether you want the response to be the records successfully written to the target system or the responses passed from the target system.

When you don't need to write data to a target system, you can use the Send Response to Source target, which passes records and responses to the source without writing to a target.

When you want to write to target systems without sending a response, you can use non-microservice targets in microservice flows.

The following microservice targets can send responses to microservice sources:
  • HTTP Client target - Writes data to an HTTP endpoint. Can send responses to a microservice source in a microservice flow.
  • Kafka Producer target - Writes data to a Kafka cluster. Can send responses to a microservice source in a microservice flow.
  • Kinesis Producer target - Writes data to Kinesis Streams. Can send responses to a microservice source in a microservice flow.
  • Send Response to Source target - Sends records with the specified response to the microservice source in the flow. Does not write data to a target system.

Sample flow

To start with a sample microservice flow, create a Data Collector flow from a sample flow and then select the Sample Microservice Flow. You can edit the flow to suit your needs. Or, you can create a blank flow and use the microservice stages in a clean canvas.

The sample flow shows how you might use the REST Service source and several Send Response to Source targets in a microservice flow:

Flow canvas

The sample flow uses an HTTP Router processor to route data to different streams based on the REST API request method and path parameter, performs additional processing, and then passes the records to the Send Response to Source targets. The targets pass the records back to the REST Service source with the specified status codes, in this case, 200 for OK. The flow error handling also passes records back to the source, with the 400 Bad Request status code. The REST Service source generates a response with the records it receives and sends it to the originating client.

The flow description contains instructions on interacting with it using the curl command line HTTP client.

Creating a microservice flow

About this task

Create a microservice flow to create a fine-grained microservice.

Procedure

  1. From the Flow Repository view, click the Add icon.
  2. In the New Pipeline window, enter a flow name and optional description and then select Data Collector as the execution engine.
  3. Then, specify how you want to create the flow:
    • Select Blank Pipeline to work with an empty canvas, and then click Next.
    • Select Sample Pipeline, click Next, and then select Sample Microservice Pipeline to design a microservice flow based on the sample flow or to review the sample flow. Then, click Next again.
  4. In the Select an Authoring Data Collector window, select the authoring Data Collector to use, then click Create.
  5. When starting with the sample flow, explore the flow and make changes as needed.
  6. When starting with a blank canvas, perform the following steps:
    1. On the General tab of the flow properties, ensure that Execution Mode is set to Standalone.
    2. On the Error Records tab, to pass flow error records to the microservice source for inclusion in the response as error records, use the Send Response to Source option.
    3. Optionally, on the Error Records - Send Response to Origin tab, configure the status code to use with the error records. Default is 500, representing an internal server error.
    4. Configure the flow using a microservice source, one or more microservice targets, and other stages as needed.