Although the service-oriented architecture (SOA) reference architecture is intentionally simplistic at a high level it, it holds some surprises when you look more closely at how components really interact. This is especially true in relation to the placement of the core elements of a business process management (BPM) capability.
We will consider some the questions that most commonly arise, such as:
- Is BPM a consumer or provider of services?
- If we choose to create an independent user interface, to what extent should it be decoupled from the BPM runtime?
- How do we retain agility in BPM while adhering to the architectural separation of SOA?
These subtleties are critical when it comes to designing solutions to fully leverage the benefits of both SOA and BPM simultaneously.
How are business processes represented in the SOA reference architecture?
A commonly recognized reference architecture for SOA is the open group's layered model shown in Figure 1.
Figure 1. The Open Group’s SOA Reference Architecture
In this layered model, "business processes" are shown in a dedicated layer to articulate their primary purpose, which is to orchestrate services that have been exposed in the layer below. Though accurate in its portrayal of the main purpose of business processes, this is a necessary over-simplified view, as high level diagrams of this type have to be, if they are to convey basic concepts succinctly.
Components in more than one layer simultaneously
The layered architecture is a logical model, not a map of the physical deployment of components. A component's position within the layers depends on the role it is playing, but some – indeed many – components perform more than one role. This means that they may in reality sit in multiple layers at once.
A simple example (Figure 2) of this would be to look at the operational systems along the base of the diagram. These operational systems are placed in the bottom layer because they are, from an SOA point of view, "providers" of function and data. Requests are made to them to retrieve or change data they hold, or to perform calculations that only they know about. However, in a modern complex environment, many of those operational systems are not completely autonomous - they also require data from other systems. In a governed SOA, requests between systems are done via calls to re-usable interfaces made available in the services layer.
Figure 2. Systems acting as both providers and consumers simultaneously
When making these requests to other systems, suddenly our operational system is, in fact, acting in a different role. It becomes a "consumer", and therefore, when performing that role, it is in the top layer of the diagram as shown in Figure 2. This means an operational system can actually appear in two different layers in the architecture depending on its role in the interaction.
So how does this relate to BPM? If we consider the runtime process engine central to BPM to be a logical component, where would we place it in the SOA reference architecture? It would seem reasonable to place it in the Business Process layer, and that would be correct, but we need to also recognize the other roles that this process runtime may perform.
Summary of the roles played by BPM in an SOA
In order to see more clearly the different roles that BPM may play in an SOA, we first need to break the BPM component up into its constituent parts. In Figure 3, we articulate the pieces as follows:
- Process runtime: This is a logical representation of the runtime component that holds the running processes, performing the orchestration specified in their process definitions.
- Process: This is a given process being managed by the runtime.
- API: This is an application programming interface that provides access to information about the running processes, such as pulling back a list of tasks within those processes that are currently waiting to be processed, and enables actions to be performed on the processes, such as claiming a task, updating its data, and completing it.
- BPM user interface: This is a BPM specific user interface that enables users to view the process data, such as viewing task lists, and perform actions, such as completing tasks.
Figure 3. The different roles played by BPM in an SOA
Figure 3 shows how the BPM process runtime is an example of a component that can be both a consumer of services and a provider. We must recognize, of course, that in any reasonably sized solution, BPM probably plays most, if not all, of these roles at the same time.
In the following sections, we'll explore what this means in more detail, and see if it is possible to represent this concept in the layered architecture.
Business process orchestration
A business process is most easily recognised in an SOA by its role as an orchestrator of services, as in c) in Figure 3. A BPM process engine can take on some of the tasks in the process that otherwise needs to be performed by people, and automates them by calling services in the service layer, improving speed, efficiency, and consistency of the process and reducing the number of human interaction points. This is the most obvious role BPM performs, and the one it is shown as performing in the SOA reference architecture (Figure 1). In this role, BPM is clearly a consumer of services, as shown in the business process layer in Figure 4.
Figure 4. BPM as a consumer of services
However, as we will see later, as with our operational systems in Figure 2, this is not the only role that the business process component plays within the architecture.
The dotted lines in Figure 4 represent the fact that there is an over-simplification in the diagram that we need to understand in more detail. Do the users really interact directly with the business process? We'll discuss this interaction in the next section.
As an aside, if we wanted to simplify the reference architecture further, it wouldn't be completely unreasonable to consider the business process when it is performing orchestration of services as a just another type of consumer. In other words, remove the idea of a separate layer for business processes altogether as shown in Figure 5.
Figure 5. Are business processes just another type of consumer?
This representation actually has some merit as it means that other non-process consumers are no longer forced to unnecessarily traverse the business process layer. Layered models are often meant to imply that interaction should only happen with the adjoining layer – you should not "skip" layers. However, consumers can, and typically do, skip the business process layer and go straight to the services layer. This representation takes that violation of the layers away.
However, for the purposes of remaining at least vaguely in line the reference architecture, we will retain the business process layer for now.
Positioning the internal BPM UI
To help us understand some of the other roles played by the business process component, we need to consider how humans interact with the process. In Figure 4, we showed a dotted line that was an oversimplification of this interaction. Let's look at that in more detail.
As a process progresses from step-to-step, the process engine assigns the next task in line to a specific group of users. They need to go somewhere to see a list of these tasks, and indeed to work on them. This user interface is often known as a "process portal". However, to be more generic, we will call it a "BPM UI" as shown in Figure 6. It enables users to view the current list of tasks assigned to someone via the groups they belong to. It also enables them to claim a task, and work on it using screens that provide just the data needed by the user, and asking for only the responses required by them.
It is this ability to provide a user with a context specific piece of work that is one of Business Process Management's key strengths: users do not need to understand how to use the underlying systems, so they need far less training in order to understand how and when to do tasks within a process. This also means that adjustments to the process can be more readily accepted since minimal re-training is required.
Figure 6. Positioning of the integrated BPM user interface
There is usually an internal BPM UI provided as an out-of-the-box component within the BPM system, which is represented in Figure 6 as the "Integrated BPM UI". In this situation, the BPM UI is fundamentally part of the same runtime component that orchestrates the tasks and services, so it is able talk directly to the process engine to query what process and tasks are available, get task and process data, and complete tasks. For many situations, this internal BPM UI provides an excellent approach for getting processes up and running quickly, and certainly during the development cycle, it provides a simple way to "playback" the current understanding of the process to business sponsors. This ability to rapidly prototype a process using the internal portal and iteratively improve it via close communication with the business users is another key benefit of BPM.
Accessing the tasks externally
In large enterprises, there are often important architecture standards in place that relate to each of the layers in the SOA reference architecture. The enterprise architecture may enforce, for example, that user interfaces must be built in a specific way, perhaps using a specific technology. They may need to comply with a host of standards around such things as the enterprise's security model, access control, presentational guidelines, and accessibility standards. In this situation it is often not reasonable to expect the internal BPM UI to be able to comply with all these standards and there is a need to build an independent user interface to surface the task lists, and tasks to the user as introduced in Figure 7.
Indeed, there may well even be a desire to embed the task lists within other systems, such that the user of those systems does not even realize they are working with tasks in the BPM engine.
Figure 7. Introducing an external BPM user interface
However, take a closer look at Figure 7 and you notice dotted lines from the external user interface to the process. There is something about this diagram that is subtly over-simplifying the interaction if we are intending to represent a true service-oriented architecture. Let's take a closer look at how the user interface is communicating with the process.
Most BPM systems cater for this common need to create independent BPM UIs by providing an application programming interface (API) that independent user interface applications can call to retrieve and update process and task information. The APIs are typically provided over common protocols such as HTTP/SOAP or REST, and maybe also over messaging transports such as JMS.
From a technical point of view, the independent BPM UI could then call these APIs directly. However, were we to let it do that, then we would be forgetting an important tenet of the layered architecture. We are discussing here how BPM fits within an SOA. We are assuming that this is an enterprise that recognizes the benefits of exposing any truly re-usable capability via the service layer, and that all requests should be forced to go via that service layer for consistency. These APIs on the BPM engine should really be seen as "task services" and "process services". These are highly re-usable capabilities across the organisation. They should, therefore, be treated as first class citizens of the service-oriented architecture, and exposed formally via the service layer.
We will consider shortly how this would look on the SOA reference architecture, but first let's fully explore the benefits of a formal service exposure.
Benefits of a formal service exposure layer
There are several advantages in exposing services formally. This is a topic in its own right, and it is at the heart of the benefits of service-oriented architecture. In brief, the formal service exposure enables the following:
- Decoupling: Consumers of the service should know as little about the implementation of the service as possible. They certainly do not need to be aware of where the service is physically located, enabling it to be changed, for example, for reasons of alterations in infrastructure, or for something more dynamic such as improved performance at different times of the day. They should also not need to know what programming language it is written in, or what operating system it is implemented on. Ideally, they should not even need to know the network protocols, or data formats used by the implementation. The service exposure layer can perform all of this decoupling and more, minimizing the effect of changes in the implementation on the consumers of the service.
- Simplicity of re-use: All services should use common mechanisms for description, and usage to other services in the enterprise such that people designing and creating new applications that consume service have less of a learning curve, and are more inspired to innovate. This can include standardizing the way that services are discovered at design time, what protocols are used to expose them, and how enterprise headers are defined and propagated. It will certainly be important to limit the number of data formats used, and it may well be appropriate to use a common data model. A common service exposure layer ensures these standard enabling re-use can be more easily adhered to.
- Standardized management: At runtime, it is possible for the service layer to be monitored and controlled using a single set of tools for such capabilities as gathering monitoring statistics, checking service level agreements, throttling traffic, and scanning for malicious usage. These things are critical if the services are to be considered predictable and trustworthy. Only if all services are exposed in a standardized way, can this level of management be provided at a reasonable cost.
- Standardized governance: How do we ensure that, from a design point of view, services follow the same rules to ensure that they are simple to re-use, conform to the enterprise's security standards, use the same access control mechanisms, utilize protocols in the same way as other services, look after its data encryption needs in the same way, and handle versioning of services via an agreed approach? Generally, only if they are all exposed via standardized services layer can we control such things, and ensure that the services play nicely within the architecture and contribute to a consistent, secure, and maintainable architecture.
If we were to expose the BPM component's capabilities formally via the services layer, they become more straightforward for future consumers to use, easier to manage, and more reliable. Put another way, if we do not expose them formally via the service layer, any new consumers that want to use them will have to become familiar with their specific security model, data format, and interaction patterns. They may need to suffer performance degradation or outages caused by other consumer's usage of the APIs.
BPM as a service provider as well as consumer
Now we see why it is reasonable, if looking at this in detail, to represent the fact that the process engine also acts in another role. It is an orchestrator of services, and thereby a "consumer". It is also a "provider" of services as a component in the operational systems layer as shown in Figure 8.
Figure 8. The process runtime acting as a provider of services
The business process engine component provides a set of services for viewing and manipulating tasks and processes, and these services are highly re-usable, so they are formally exposed through the service layer.
For illustration purposes, we have added the dotted lines in Figure 8 to show that the process runtime providing the API communicates with the Process Engine performing the orchestration, for example, task completion requests that allow the orchestration to continue. This dotted line appears at first to break some fundamental architectural principle by calling upward through the layers and without formally going through a service layer component. However, remember that the Process Engine in the Operational Systems layer and in the Business Process layer is in fact the same component. The dotted lines are actually a perfectly acceptable internal communication within this single component. These dotted lines are, therefore, in reality probably best removed from the architectural diagrams to avoid confusion. They are just shown here to explain what is really happening beneath the surface.
Design points relating to an independent BPM UI
There are certain implications of having an external BPM UI that is created and maintained in a separate technology to that of the process engine and separated by a formal service layer.
How do we retain agility?
Fully separating the user interface from the process probably means that the user interface will then be written by a completely separate team, and probably in a different technology. Unless there is very good collaboration between UI and process teams, this will slow down the pace at which iterations of the process can be prepared, tested, and deployed. However, the greatest rate of change of the process and UI is at the early stages of process discovery, so a it would be wise to continue to use the internal BPM UI for playbacks at this early stage, and for as long as possible until the "wireframe" of the user interfaces, and the data model for the process, have stabilized.
Note that you should not make any assumptions that the wireframes created during this early phase can be directly re-used in any way when you move to the external user interface. They should focus on what data is displayed, how it is grouped, and how screens are roughly navigated, all of which should be relatively swift to recreate in the external UI technology later. Subtle styling such as color, format, and precise positioning should be done when developing the final external user interface. This separation of "wireframe" design from detailed presentational design of the UI is commonplace in many development methodologies and helps to focus efforts on the requirements that matter in the early phases, rather than getting side tracked by unnecessary details.
Figure 9. What to capture at each phase in relation to BPM
This progressive capture of detail is illustrated in a basic form in Figure 9. If at all possible, we aim to retain the benefits of the internal BPM UI through the "process design" phase, where significant refactoring is likely to be happening. Ideally, the core interplay between user interface and process should be bottomed out in this phase. In the implementation phase, most of the effort is focused on independent concerns of the precision in the user interface, and the detail of the integration with operational systems. There is always some leakage from changes affecting all layers, but retaining agility during the design phase by using the internal BPM UI goes a long way to help stabilize the subsequent phases.
The user interface required for a single user to complete a single task often in itself comprises of several separate pages. The user must navigate through these in a certain order, often dependent on the data they enter. This movement between pages while performing a task is called page navigation. It is most certainly not a business process, although it can, and often is, represented using a similar notation. It is inevitably a common mistake when modelling processes to confuse the two, and there are many modelling best practices to help avoid this mistake. Page navigation accidentally modelled within a process is relatively easy to spot, as you will see multiple tasks line in the same swim lane. Generally, subsequent tasks in a process diagram need to change lanes, passing control to a different team. If the work remains in the same team and swim lane, question why it is not all part of the same task.
The two types of modelling look similar and many BPM process design tools can capture pagenavigation alongside the process diagrams, and often allow for page design as well, as shown in Figure 10.
Figure 10. Process, page navigation, and page design
If we decide to separate the graphical user interface from the process, the pages belong in the user interface technology, and the process remains with BPM. However, we need to decide where to place the page navigation logic. It is tempting to conclude that since the notation is similar, the page navigation is part of the process definition. However, this would be to misunderstand how tightly coupled pages are with their associated navigation.
Consider the following, illustrating how close the page design and page navigation really are:
- Navigation logic: Where the page navigates to is likely to be highly dependent on the data entered on previous page.
- Data dependencies: Data on one page may affect data on other pages in ways such as field visibility and filtering lists.
- Performance: Responsive user interfaces are critical to the front line staff, and performance is key. Pre-fetching data for the next page is done if the user interface knows enough about where the user is likely to go next. Pages themselves may need to be pre-loaded.
- Data models: It is likely that the pages will want to hold the data in a data model that represents the data in a way that is simple to render on the pages, with data grouped and represented in the same way as it is across the pages. Furthermore, the pages are likely to make use of re-usable UI components that hold the data in a specific form. The page navigation needs to work directly with this data model if it is to avoid continuously translating between two models. It is often the case that this data model is significantly different than the model used by the underlying process.
Now consider these points from the process point of view:
- The process "doesn't care" how a task is achieved, how many pages there are, how they are navigated, or even how long it takes. It cares only about what data is required to achieve the task, and what data it expects back when the task is complete.
- Process design tools deliberately simplify user interface design such that they are done alongside process design, improving the value of the playbacks. However, process designers do not typically have extensive training in user interface technologies, or in user interface design. User interface design for front line users, such as those in customer facing scenarios, is a separate and highly specialized skill. As you would expect, the design of a high quality user interface, the page, and the page navigation are deeply intertwined.
Therefore, it is clear that if an external user interface is to be created to work the tasks, it needs to contain both the pages and the page navigation as, shown in Figure 11.
Figure 11. Separation of concerns for an independent user interface in BPM
This separation of concerns ensures that the appropriately skilled teams can do what they know best, and that the only thing that needs be agreed between them is the input and output data for the task in hand.
Security model translation
It is likely that the security mechanisms enforced in the service layer are different from those of the BPM Engine. Indeed, this may well be one of the reasons the service layer is present – to provide a consistent security model throughout the enterprise. However, although most security model translations can be overcome within the service layer, they are often technically challenging first time around, and this is considered a significant "spike" that needs to be tackled early to reduce risk to the project.
Data model decoupling
Do we accept the data model provided by the underlying BPM product's API? If the product API does not conform with our protocol and data format governance standards, do we then expose the product API's data model directly, or do we introduce a service layer data model that is independent of product? This is a complex issue, but some of the obvious ramifications are that if we use the product's data model, we will suffer if we change the product, or potentially even upgrade it to a new version. However, if we choose to introduce a new data model, then that will have to be designed, implemented, tested, and kept up-do-date with new product capabilities. This is a non-trivial task, and it introduces its own risks. Typically, products have a well-thought out API, and since it is a publicly specified interface, the vendor will already be required to manage issues, such as versioning and backward compatibility when the interface changes. It is often better, therefore, not to translate the data model unless absolutely necessary, and to focus the decoupling on more non-functional concerns. However, there are some special cases where we may choose to perform functional translation of the API and these are covered in the next section.
Exposing the process via a “business” service
There is one further type of interaction with the business process that to consider. To date, the services we have been exposing for re-use have been essentially a "generic" APIs, provided by the process engine for directly administering task and processes. These are clearly flexible APIs enabling you to perform almost any action required to a task or process. However, they have a number of disadvantages:
- To use them effectively, a designer or developer must intimately understand the concepts of tasks and processes, and know how to navigate them.
- The granularity of these APIs is often quite low level, and it may take multiple requests to achieve an apparently simple objective such as completing a task.
- The business data retrieved from a task, or required to update it, is not known to be this generic API. Therefore, it is not possible for a consumer of the service to inspect the schema of the API in order to understand how to populate it, or retrieve data from it. Separate documentation or schemas will have to be prepared to explain the specific use of the API for each task in each process.
There are circumstances where we want to enable consumers to perform actions on a process in a more "business" oriented way, without having to understand the intimate details of the process engine. This can be done by providing a business specific service, which is then mapped to the appropriate lower level requests on the generic API. We have shown this in Figure 12, where the user interface calls a submitLoanApplication() method, which takes specific and well-defined parameters such as the LoanApplicationForm parameter. This is then translated by a service component into the relevant generic operation startProcess() on the API for the BPM runtime's API.
Figure 12. Exposing "initiation of a process" via a business service
Once again, for completeness we should mention that the Process Runtime in the Operational Systems layer is, in fact, the same component as the Process Runtime in the Business Process layer. Therefore, the call between them shown with a dotted line is, in fact, an internal request within the BPM component. This level of detail helps us to illustrate what is really going on across the layers of the service-oriented architecture. But, as mentioned before, it may be inappropriate for day-to-day design diagrams.
There are two different common examples where a process can be exposed as a business service. Figure 12 describes only the first one, initiation of a process.
- Initiation of a process: Many business processes begin with an initial data capture phase, where a significant amount of data is collected, often from the actual customer via an online web site. Examples include applications for a quotation for an insurance product, or applying for a bank loan, or populating an online shopping cart before submitting it to be processed. The UI from an internal BPM UI is typically not the appropriate mechanism for this data capture. The user often does not even know that they are taking part in a process, so a dedicated independent UI, or even a device application, is likely to be used to collect and ultimately submit these data. The data submission services are often highly re-usable, and creators of such UIs or applications that consume them have no need to understand the intricacies of the process engine. They also benefit from an interface schema that clearly defines the business data required so they know what they need to populate for a valid request.
- Correlating with a running process: Mid-way through a process, there is a need to pause waiting for an event to happen. Typical examples are waiting for the receipt of a document from a customer, or the arrival of a manual quotation from a supplier. Again, these events are likely to be captured by separate applications or user interfaces external to the process engine, and the writers of those applications may only know that they must publish an event, but not who will be listening for it. It is likely that they do not even know that they are delivering it to a process at all. They certainly do not want to understand which task in which process they should be passing the event to. In this situation, a business service is exposed by the service layer to be called when the event happens. A service component then translates that into an event for the BPM component. The subtlety here is that the event must correlate with the correct running process instance. Most BPM engines have a correlation capability for incoming events exactly for this reason.
In both cases, a business specific service is required that specifies the valid business data model. This is then transformed into the relevant requests to the lower level generic API on the process engine.
Clearly, this approach has advantages for the consumer of the service in terms of simplicity. The consumer is completely and functionally decoupled from the way in which the service is implemented, and is not even aware that the service is provided by a process engine. On the other hand, this results in extra work defining the business service for every process that needs to be triggered or for every event that meets with a process rather than using a generic API. In essence, the more re-usable the business service is, the more beneficial it is to use this technique.
With strict SOA reference architecture layers as a starting point, we accepted that it is reasonable for a component to appear in more than one layer, depending on the role it is playing. This enabled us to make sense of the dual role that the BPM component plays, and move on to the more subtle aspect of how the interactions really work.
We have noted that there are important decisions to be made when architecting BPM within an SOA. The primary three can be summarized as:
- Does the internal BPM UI suffice for our production needs? If not, how long through the development lifecycle can we retain the use of it and the agility that it brings to process design?
- If we choose to create an independent user interface for processes and tasks, should the BPM API be exposed formally through the service layer to improve decoupling and management of the interaction?
- Are there are circumstances where business specific services (as opposed to the generic task or process services provided by the BPM API) should be provided in order to further decouple the consumers from the knowledge that they are interacting with a BPM implementation?
None of these questions have simple answers, but we have discussed the pros and cons of each. We can certainly say that, as a general rule, the larger and more complex the enterprise, the more formal separation of concerns provided by the SOA reference architecture will need to be followed if the overall landscape is to remain manageable in the future.
The author would like to thank the following people for their help in preparing and reviewing this article (in alphabetical surname order): Andy Garratt, Eric Herness, Alan Hopkins, Brian Petrini, Rob Phippen, John Reynolds, Paul Smith, Claudio Tagliabue, and Richard Whyte.
- SOA Reference Architecture
- Video: BPM and SOA: Better together
- Process-oriented modelling for SOA, Part 1: A technique for process decomposition
- IBM Redbook: Business Process Management Enabled by SOA
- BPM Voices: Where does BPM end and SOA begin?
- developerWorks BPM zone