Topologies for the Data service connector in DataStage jobs

Using the Data service connector in jobs implies a set of constraints and guidelines. The service-oriented architecture (SOA) platform supports three job topologies for different load and work style requirements: jobs with a data service input stage, jobs with a data service output stage, and jobs with data service input and output stages.

The design of a job determines whether it is always up and running or runs one time to completion. All jobs that are exposed as services process requests on an ad hoc, 24x7 basis.

Make sure that the column names in your DataStage® jobs do not contain a numeric digit followed by lowercase letter, such as 2b. This type of column name will cause your jobs to fail.

A service accepts requests from client applications, mapping request data to input rows and passing them to the underlying jobs. A job instance can include database lookups, transformations, data standardization and matching, and other data integration tasks. A job instance can then return output rows that can be mapped to service response data and sent back to the client.

Data service connector as an input only

Topology I uses new or existing jobs that are exposed as services. A job starts on demand and continuously runs until it receives a stop request. The job typically initiates a real-time process that does not need direct feedback on the results. It can accept job parameters as input arguments. Topology I jobs have the following characteristics:

Start times: The elapsed time for first starting a job, also known as latency, is high. This factor contributes to a low throughput rate in communication with the job service and RabbitMQ service.
Input: An information service that is based on a job with the Data service connector as an input can use job parameters as input arguments. This type of service returns no output. If the job ends abnormally, the service client receives an exception.
A job can have only one Data service connector input stage. The Data service connector input stage supports one output link. Its table definition maps to the input arguments of a service operation, that is, the input data arguments of a web service request. Input values can consist of an atomic value (single column), a structure (multiple columns), or an array of structures (multiple rows).
Defining stages: Jobs with passive stages that have both input and output links are not eligible to implement service operations. The job that has a data source stage with input and output links cannot be used to implement a service operation. If you need a data source stage with both input and output links, do not use a Data service connector input stage.
Also, a database query with results that are rerun on an output link is run only one time, when the job instance is started. It is not re during every service call request. If you want to extract data from a data source, use a reference link to perform the lookup.
Always-on behavior: A job that conforms to Topology I is always running. A job instance handles multiple requests during its lifetime. If the job stops, the service client receives an exception.

Data service connector as an output only

Topology II uses a new job and adds an output stage. A job starts on demand. The Data service connector output stage is the exit point from the job, having one input link, and returning one or more rows to the client application as a service response. Its table definition maps to the output arguments of a service operation, that is, the return value of a web service operation. Return values can consist of an atomic value (one column), a structure (multiple columns), or an array of structures (multiple rows). These jobs typically initiate a process from a real-time process that requires feedback or data from the results. It can accept job parameters as input arguments.

Data service connector with both an input and output

Figure 3. Example flow of Data Service connector as both input and output

In Topology III, jobs use both a Data service connector input stage and a service output stage. The input stage is the entry point to a job, accepting one or more rows during a service request. These jobs are always running. This topology is typically used to process high volumes of smaller transactions where response time is important. It is tailored to process many small requests rather than a few large requests. Topology III can accept job parameters as input arguments.

Topology III jobs have the following characteristics:

Start times: The elapsed time for first starting a job, also known as latency, is high. This factor contributes to a low throughput rate in communication with the job service and RabbitMQ service.
Input and output: A job can have only one Data service connector input stage. The Data service connector input stage supports one output link. Its table definition maps to the input arguments of a service operation, that is, the input data arguments of a web service request. Input values can consist of an atomic value (single column), a structure (multiple columns), or an array of structures (multiple rows). The maximum input size is 2 MB. Larger inputs send an error message to the client application.
A job can have only one Data service connector output stage. The Data service connector output stage supports one input link. Its table definition maps to the output arguments of a service operation, that is, the return values of a web service request. Return values can consist of an atomic value (single column), a structure (multiple columns), or an array of structures (multiple rows). The maximum output size is 2 MB. Larger results are truncated and send an error message to the client application.
Defining stages: Jobs with passive stages that have both input and output links are not eligible to implement service operations. The job that has a data source stage with input and output links cannot be used to implement a service operation. If you need a data source stage with both input and output links, do not use a Data service connector input stage.
Also, a database query with results that are rerun on an output link is run only one time, when the job instance is started. It is not re during every service call request. If you want to extract data from a data source, use a reference link to perform the lookup.
Always-on behavior: A job that conforms to Topology III is always running. A job instance handles multiple requests during its lifetime. If the job stops, the service client receives an exception.