Implementing Worker Tasks

Worker tasks as opposed to backend routines, are run by the Optimization Server. The execution of the task takes place in a worker microservice: each worker task is associated with a worker microservice, although a worker microservice can host several worker tasks.

Two types of workers are available:

  • Java worker tasks: those workers encapsulate an optimization engine that must be packaged in a Java library. For more details, please refer to Section Implementing Java Worker Tasks.

  • Python worker tasks: those workers encapsulate an optimization engine that must be packaged in a Python library. For more details, please refer to Section Implementing Python Worker Tasks.

The execution of a worker task happens as follows:

  1. Data is prepared by the Data Service and is sent in a CSV format;

  2. The CSV format is received inside the worker;

  3. An adapter is used inside the worker to transform this data into a format that the processing engine can understand;

  4. The task action performs its work (optimization, etc.);

  5. An adapter is used to transform the results of the task action into the CSV format;

  6. Data is sent back to the Data Service for synchronization.

As a consequence, the implementation of a worker task includes:

  1. Coding the task actions in a class (or a set of classes) located in a Gradle project under the processing folder.

    The processing folder is the place where you will put the code you write to implementing your processing logic. To implement your task, you will thus write one or several classes, and package them as a Gradle project under this folder to produce a library that will be used by the worker microservice responsible for executing the task.

    The processing folder initially contains several projects that can be used as a starting point for your development. They are:

    Test this implementation using JUnit tests and data that is compatible with the specific type of Worker. For instance, engine contains a JUnit test class that illustrates how to run unit tests against your code, for example, with data snapshots can be used for Java worker tasks. For more details, please refer to Sections Saving Data Snapshots, Implementing Java Worker Tasks and Implementing Python Worker Tasks.

  2. Generating a worker microservice and add the task Optimization Server description in YAML format to this microservice. Alternately, add the YAML description of the task to an existing worker microservice.

    The Platform includes a dedicated code generator for worker microservices.

    Three predefined workers are generated:

    • engine-worker and checker for Java

    • python-engine for Python

    Each generated worker is a new service that will register itself with the Optimization Server master service. It can be started as a Java process as all the generated services or as a Docker image.

    It is possible to generate additional workers. For more details, please refer to Part Getting Started. For more details on how to author a worker microservice, please refer to the Optimization Server documentation, Chapter Workers .

  3. Adding the adapter code in the worker code to process the input and output of the task actions as well as the call to the processing implementation.

    Each worker task is described by an entry in the worker.yml file, located in the resources of the associated worker microservice. The code generator for the engine and checker workers also generates this YAML section. For custom workers, however, you need to write the worker.yml file yourself. For more details, please refer to the Optimization Server documentation, Chapter Workers .

Once these steps are completed, the execution of a worker can be triggered from a scripted task, as explained in Chapter Understanding the Execution Service.