Integration Workflow Processing and Performance

This page has not been liked. Updated 10/20/15, 6:13 PM by agenaTags: None

Increasing performance of integration operations by disabling workflows that are not used by that operation

When doing integration operations like importing large numbers of floor and space records using Data Connect (DC) you can increase the throughput by understanding what workflows are being executed as part of the process and disabling those that are not needed given the nature of the operation (creating many records with known good data).

One way to find out what workflows are being executed is to enable all workflow instances to be recorded and then inspect them after going through the process once. However, that can be very tedious and it's easy to miss many workflows that are executed as side effects of creating and associating records. In addition, enabling workflow instance recording can cause significant overhead to the process and the entire system so try to avoid it except when debugging some part of the processing. In general you want all instance recording disabled (you can set that option in and temporarily override it if needed using the Administration Console - Workflow Agent page).

Instead of instance recording, if you want to know what workflows are executing and why, enable the workflow performance logging (both SYNC and ASYNC) and collect the performance.log file from the server (the process server running the workflow agent). You can process the performance.log using the Workflow Performance Analysis Tool to get an interactive view into everything that's running, why it's running, and how long it is taking.

You can get the tool from the IBM ISM site:

Also, if you do the integration processing on a regular basis, look at using the 'Integration' flag in workflow start conditions to cause workflows to be skipped rather than temporarily retiring them.


Increasing performance by limiting the number of workflow agents

The point of running multiple workflow servers is to allow workflow processing to be done so that it is fair to all users, not necessarily to increase the throughput of the number of workflows done.

Adding more workflow agents to an environment can slow down processing, and cause undesirable results, if workflows are not written with multi-threading in mind.

It is best practice to assign secondary workflow agents to specific power users that tend to run more workflows than a normal user.  If the secondary workflow agents are left wide open, a set of workflow instances are picked up in parallel, and some can be processed out of order. It is important to know that increasing the number of threads on a single process server results in higher throughput than splitting the threads across two servers.  Typically the bottleneck of performance in an environment is the database server rather than the process servers.

If you already have a system that is deployed with multiple workflow agents, consider either:

* stopping the secondary agents, and increasing the threads on the primary workflow agent server to be the sum of the threads across the other servers

* or restricting the secondary agents so that they are exclusive for the set of power users.