You read it in the business magazines that cover information technology - Business Process Management (BPM) is easy, even a child (mind you, a "smart" one) can do it. Business Process Modeling Notation (BPMN) is "just" flowcharting workflows - how tough can it be? But, it is not that simple in many cases. Sure, technology has made it easier than writing Java™ application code to orchestrate a process, and then building a web front-end for the application. But, to think you can do BPM in an ad hoc manner is often a huge mistake - and one that continues to plague BPM initiatives.
It is important to note that some process applications can be implemented easily, but all BPM applications require the discipline and rigor that are the fundamental foundations for the development of business applications. In fact, you can argue that BPM's business-centric user population actually acerbates the need for this architectural rigor. Many business users do not understand the underlying complexities associated with application runtimes and IT needs to provide oversight to ensure the applications are optimized. In short, just because BPM appears to be "simple" - organizations still need a sound architecture to ensure robust applications. We will discuss the key architectural aspects that are critical to building an optimized BPM application with references to case studies to better craft BNPM solutions that are architecturally sound.
The benefits of BPM
First, it is important to clarify that BPM is a critical competency for organizations in general - the ability of an organization to use BPM to optimize manual or swivel-chair processes is key in many areas such as regulatory management, back-office transactions, claim management, case management requirements, and many other key business application areas. BPM can bring great benefits to an organization in these areas by providing the following capabilities:
- Automating workflow and decision making
- Reducing errors and improving consistency
- Standardizing processes across geographies and lines of business
- Leveraging existing systems and data
- Enabling the monitoring of key business events and initiating actions when events surface
- Providing real-time visibility and process control
At the same time, it is important to note that BPM solutions are software applications - they require a sound design to provide an effective solution. However, all too often we find that organizations forget the basics of software design when developing BPM.
We hope that readers of this column have a basic understanding of BPM development methodology. This area is covered in a number of the references listed at the conclusion of this column for readers who need more information.
First things first - requirements discovery and definition
As software practitioners, we recognize that requirements are the first step to designing effective solutions. However, we find that organizations embarking on BPM initiatives often do not consider requirement discovery and definition as a key step in BPM application design. In most cases, the requirements are not done with any rigor and we find that requirements are often poorly documented. Finally, we find that the business is not actively engaged with IT to determine if the requirements are feasible. Bottom line - requirements are fundamental to success with BPM.
Let's start with an example. We are currently working with a client that is deploying multiple international BPM applications. The unfortunate observation we found during an application review - there were no formal requirements. As a result of the requirements gap, there was no formal test plan and the deployment and operational management processes were basically undefined. The key technical sponsors have read some books and attended briefings on BPM, but they have limited understanding of the technical aspects of the solution. Worse yet, these are the technical stakeholders. There needs to be a better way.
The key point in this use case is that the lack of effective requirements definition will lead to huge challenges in building BPM solutions. Though there are ways to circumvent the incumbent misery associated with traditional requirements discovery - there is never a good reason to bypass formal requirements definition. When the aforementioned client was asked about the gaps - they replied "well - they are only simple workflow applications". This attitude is downright frightening, but unfortunately we see this quite often in BPM development projects. Organizations simply do not address requirements for BPM applications in a consistent and formalized way.
We all remember the glory days when we were using the "magical" client-server 4GLs, such as Visual Basic and the like. They continue today. IBM® was called in to assist a client troubleshoot their new application they proudly built (without any IT involvement) on Visual Basic for Applications in Microsoft® Access. They built the application, populated the database, and stored it on a shared file server where 20 people tried to access it at the same time. They called because they were surprised when the data got corrupted and could not figure out how that had happened. Sure, they built the application by themselves, but at what cost? And, more importantly, had they involved IT and done the appropriate design work - these issues would have been mitigated (or perhaps avoided altogether).
There is a prevailing impression that BPM development means that you can bypass normal development approaches. With the advent of agile development and rapid prototyping, you can certainly circumvent some of the hurdles in prolonged analysis and design approaches. But, to think that these requirements simply vanish is a misconception that continues to hinder development of quality applications. For some puzzling reason, the business "expects" applications to be developed akin to a sausage factory. True enough - if you get the fundamentals of design and development in place, you can develop BPM solutions with a factory approach. However, you have to incorporate requirements gathering and architectural rigor in the development up front - you cannot ignore it. We find that the obfuscation of this basic approach to development causes business user frustration and development rework and detracts (and often destroys) the benefits of BPM. Figure 1 shows the key approach we often find in terms of requirement gathering (or lack of requirements gathering) for BPM.
Figure 1. Classical requirements gathering
The lack of requirements gathering early in the design process is a key factor in destroying the promise of BPM. Without defined functional and nonfunctional requirements, you cannot design nor deploy a good solution. This is a foundational precept for all application development and BPM is no exception.
There are new and innovative ways perform requirements gathering for BPM. We encourage clients to use the IBM Blueworks Live solution for process discovery. In fact, the IBM Software Services teams use IBM Blueworks Live (BWL) when working with clients to "discover" their business processes. This is a key part of our services delivery for BPM - and we encourage clients to use BWL for similar reasons. Blueworks Live has excellent features that allow organizations to capture system and technical requirements in line with the business process requirements. The BWL solution provides a collaborative web approach so that business and technical users work together to discover and define the key BPM requirements. This approach allows the stakeholders to define and subsequently track the business and technical requirements that are the foundation of the business process. BWL provides a great way to capture key milestones and tasks in the process, to define these attributes interactively and to provide traceability as the project progresses. But, regardless of whether or not an organization chooses to use BWL - requirements discovery and definition is a key step in the development of BPM applications.
As Lewis Carroll remarked in Alice in Wonderland, "If you don't know where you are going, any road will get you there". But, that is not an end in itself. It is quite likely you will "walk" a long distance only to find yourself an even longer distance from where you should be. This key point reinforces the key premise that architecture and BPM are linked. Just because BPM is seemingly simple (or so marketing tells us), you can't cut corners. You need to invest the time upfront to ensure a successful implementation and deployment of BPM applications. This column focuses on the need to consider architecture as a key aspect of BPM application development - to avoid this will nearly guarantee a suboptimal application and, in many cases, a failed application.
Architecture and BPM - "Like peas and carrots" as Forrest Gump used to say
Perhaps the best place to start our discussion is by defining the attributes of a good BPM application and to describe why architecture is critical to the implementation of BPM applications and the long term success of BPM initiatives, especially at the enterprise level. Just one unsuccessful BPM implementation will likely derail enterprise BPM initiatives as business users will be jaded on the potential benefits of BPM.
So back to the original question - what is a good BPM application? Actually, this is a tough question, and there is not a simple answer. Many factors contribute to a good solution (and it is not just a BPM issue). We believe the key litmus test for a good application is satisfying (or hopefully exceeding) the business user expectations. Sounds easy, huh? Not really - this focus on the target end user underscores the need for collaborative and agile approaches to solution design, development, and deployment. And these requirements are even more critical when we consider BPM. Bill Gates once pointed out that "automation does not fix errors, it just makes them happen quicker". Without a concerted effort to design the right solution and to implement the solution with a good architectural foundation, BPM will compound (rather than solve) the problem at hand.
We will not cover collaborative development approaches for BPM in this column. There are a number of references at the end of this column that focus on these areas, but we need to state unequivocally that this approach provides the foundation for our ongoing discussion. Without shared business-technical ownership and an agile BPM approach - the rest of our premise is, in a word, moot.
Bottom line - business stakeholders must be engaged at the beginning of the BPM development process and throughout the life of the BPM application. Without this, there is no shared commitment and, as a result, the definition and implementation of the functional and nonfunctional aspects of the envisioned solution will be handicapped from the onset. It is rather like playing darts with a blindfold over your eyes and hoping you hit the dartboard. Business and technical alignment and agreement on requirements is the key first step of developing a BPM application.
Let's return to our original premise. Although the IT "marketplace" may convey the impression that BPM is easy and simple, we should note that BPM is not easy. The complexities of a BPM implementation have their roots in the architectural challenges in building composite applications - these include orchestration of user interfaces, integration with systems of record, managing nonfunctional aspects, such as transactional requirements, availability, governance, performance and scalability, and so on. BPM development is certainly not easy. Additionally, the visibility of BPM solutions to end users makes this architectural focus even more important. Business stakeholders in a BPM implementation have key expectations. Without a laser-focused approach to satisfying those expectations, the project will be viewed as non-optimal and will be labeled as a failure. This inability to meet user expectations has downstream effects in generating a negative perception of BPM overall.
We can use an iceberg metaphor to visualize this - much like a portal application. Users are focused on the user experience as they interact with the application. Users of BPM applications are concerned with doing their job quickly and successfully - the BPM application hopefully enables this. But, it is more than just the user interface. Although the applications may "look appealing", this is just the "tip of the iceberg". The true measure of a successful BPM effort lies in the ability for business users to be productive with the application and to find value in the solution. These are the qualities that sit below the water level and represent the "hidden attributes" of the application. This is where the "hard work" comes in - the rigor and discipline of designing an application based on sound architecture principles and satisfaction of the requirements.
During a recent engagement, we were working with a group of business users that did not have any direction in terms of an architectural approach. This is why it is critical that business works with IT to develop BPM solutions. Currently, the business users employ multiple systems where they input and update a large amount of data. After quick demonstration of the basics of an IBM BPM solution, they said (almost in unison), "This is great, but this BPM thing cannot be yet another tool where I have to enter data." This observation reveals that business users also understand the basic architectural elements. In effect, the business users want to ensure that BPM will integrate with all of those existing backend systems that they use today. If the BPM application is simply providing yet another UI that they have to learn, there is no value in the approach. Although today's BPM tools make it very easy to build user interface components, this is not enough. The solution must provide an ability to optimize the way that the business interfaces to the existing systems and provide a more comprehensive approach to performing work. Otherwise, it is just another tool to weigh down productivity.
We should note that BPM applications are the "ultimate" composite application. They act as the key layer between service consumers and the underlying services and enterprise applications. As a result, they comprise a translation layer mapping user interactions into backend systems via orchestration and choreography. In addition, they manifest cross cutting concerns such as integration functions, quality of service requirements, data architecture, and governance aspects. This is shown in Figure 2.
Figure 2. BPM and the SOA solution stack
Case closed - BPM is not simple. Even the simplest workflow application needs to be developed with a disciplined and rigorous approach. And, the sad fact is that most BPM applications are neither designed nor implemented based on this fundamental truism.
The recipe for "good" BPM applications
In the article Evaluating BPM applications: BPM design reviews and Rubik's Cubes, Scott Simmons discussed ten key areas for business process "goodness", which need to be covered as part of an application review. We have listed them below in no particular order of importance:
- General BPM solution design (including business architecture, process ownership)
- Process modeling/design approach
- Information modeling
- Decision services
- Event management
- Integration services and interfaces
- User interface design
- Architectural aspects (such as nonfunctional requirements)
- Infrastructure and deployment considerations
- Governance aspects
That article discussed the key aspects in each of these areas. We will not bore you with the all of the details here (though you are urged to read the article if you want to know more in this area). The key takeaway is that each of these elements plays a key role in the successful implementation of a BPM application and needs to be considered in the overall design of the application.
At the same time, we need to state that some BPM solutions will require a greater focus in specific areas. For example, the choreography of a solution that integrates with SAP will have a greater focus on SAP integration, whereas another solution that focuses on payment reconciliation will have emphasis on transactional and performance issues with backend payment systems. But, in both cases, all ten of the aforementioned areas need to be considered in the design of the overall solution and need to be reviewed and assessed prior to going live with the application. Many of the factors within these 10 functional areas are "invisible". In fact, most of the time, they are not directly visible to the end user. But, cutting corners on any of these areas will likely cause considerable re-work over the application lifetime. If they are not addressed early in design, the problems will be compounded. And this is not just a BPM issue, though BPM is our focus here.
Consider an example of a company working to provide improvements for its call centers. At first glance, it might seem like BPM is a natural fit for this type of application and we find that our clients often focus on rebuilding their current call center application (or parts of it) using BPM. This approach is often an anti-pattern. BPM's strength in this application should be the ability to launch and manage new process instances to support call center operations. The value of BPM lies in the ability to manage the follow up work after the caller hangs up. But, we find that clients want to use BPM to support the call center interaction with the client, which is often more appropriate for a business rules management solution to guide the call center representative through the steps needed to fulfill the caller's request. Back to architecture - it is important to determine the requirements, and then to define the various use cases or user stories that a call center rep may take through the application. If we use BPM to solve the call center interaction aspects, we may end up with yet another monolithic application that provides little value over the original application.
Therefore, it is important to drive BPM solutions from an understanding of the requirements and to ensure that BPM becomes the platform for process management and is not just "another application". In the call center example, BPM should provide the solution for ensuring the work is allocated to specific roles and employees without management needing to manually distribute the work, enabling escalation and tracking of process tasks and providing a view of outstanding and historical tasks to support process optimization over time.
At the same time, we need to understand the architectural aspects of this solution, there may still be a significant amount of work to be done under the covers to provide integration of the solution with key infrastructure components. In many cases there are underlying systems (LDAP, business rules, and databases) that contain the necessary information to support the process applications. Those systems can be highly visible, high volume, and critical parts of the infrastructure. When these systems are not well integrated with BPM, solution performance and availability will suffer, and the application will not behave as intended and will often lead to the dissolution of the BPM project. We find that BPM technology continually gets the blame when infrastructure components are not addressed within the solution, and normally we find out about this after the fact. For the remainder of this column - we will focus on architectural aspects and integration services and interfaces - as these are two of the key areas where we see client challenges as they pursue more complex BPM solutions.
Figure 3 shows the key components in the IBM BPM solution. The top portion shows the key BPM consumers such as mobile, standalone and portal-based user interfaces, runtime components such as core and advanced BPM functions, the linkage with the Process Center to provide governance and visibility of BPM assets, and a set of developer and management tools to expose technical aspects and development functions.
Figure 3. The IBM BPM solution architecture
It looks simple, but understand that this diagram does not really illustrate the integration and non-functional requirements. More importantly, a "fool with a tool is still a fool", even the best operational and developer tools will not compensate for bad design. It is clear that BPM applications span a wide range of solutions. We can do a high level review of typical BPM solutions and classify them into low complexity, medium complexity, and high complexity. We can point out some "blinded" examples:
- Low complexity: One financial services client provides a solution to satisfy LEI (Legal Entity Identifier) requirements. The process is triggered from an external event, but it is self contained. The solution provides basic query functions into backend systems, but consists of simple user interfaces and does not have stringent nonfunctional requirements.
- Medium complexity: Another financial client provides an interface with their payment processing that enables routing and disposition of suspect payments as part of an Anti-Money Laundering compliance. The application enables users to determine whether such suspect payments should be processed or reviewed based on key metadata. The application has a significant integration component, but the overall workflow process itself is straightforward. Due to the integration requirements with product processing systems and nonfunctional aspects, this application can be characterized as "medium complexity".
- High complexity: One key financial client has deployed a solution for corporate account opening services. The application interfaces with five backend systems with detailed requirements for transactional support across these systems. The solution consists of a large rule base (> 50 rules) and has a large number of complex user and administrative interfaces.
Though all of these applications require architectural design and review, the medium complexity and high complexity solutions require a higher level of oversight and design to ensure that the architectural requirements can be satisfied. Both applications have significant integration requirements and non-functional aspects that need "extra attention" when performing the design activities. We have found that these types of solutions are very challenging as the components "below the surface of the iceberg" are not normally well-defined. Often, we find that these types of applications consider the transactional, performance, availability, and security concerns late in the design process versus early as part of the overall solution definition.
Recently, we worked with a client building an account management application. The solution implementation needed to interface to seven different enterprise systems to integrate customer and account information on both mainframe and distributed tiers. The application required integration with the enterprise content management store to support the storage and retrieval of customer account application forms. Although the solution was predominantly BPMN in nature, there were a significant number of BPEL-based Straight-Through-Processing components requiring the use of Advanced Integration Services (AIS). The solution had numerous nonfunctional requirements in the following areas:
- Security: There are multiple entitlement requirements in terms of which roles can perform specific operations in the account opening process. The solution needs to integrate with an existing LDAP directory and provide role assignments based on directory metadata.
- Performance: Application volumes are moderate, but the performance of specific interaction steps, such as querying and updating backend systems, is critical to maintain user productivity during the application execution. As a result, analysis of the SLAs with backend systems and optimization of the integration between the BPM solutions and the enterprise systems is critical.
- Availability: The BPM solution needs to be available in a 5X24 hour configuration to enable a "follow-the-sun" approach for account opening. The solution needs to be accessible globally so that different work centers can assist in activities in the process.
Although the client realized these challenges in the early design process, unfortunately they were not factored into the design. Six months after the BPM solution development started, numerous questions surfaced that resulted in significant rework in areas, including solution topology, integration services, and overall nonfunctional approaches. This rework caused churn in the project implementation and impacted solution delivery by four months. The business stakeholders were not happy and viewed these issues as product flaws and reflected poorly on the use of BPM for the project. In short, this was not a good situation. Had these questions been raised earlier in the design, many of these issues would have alleviated. We will use this scenario as our representative example of how to design (and how not to design) an optimized BPM solution.
Architecting BPM the "right way"
There are key principles that can be used to minimize the downstream issues and ensure we define the right infrastructure based on both functional and nonfunctional requirements. It is important to note that challenges always arise due to unanticipated requirements encountered during design and implementation. The focus for the remainder of this column is to discuss approaches to minimize the "churn" through planning and design rigor.
We recognize that architectural skills are not always present in BPM development teams. We recommend that organizations needing additional skills in the architecture domain obtain training and mentoring from IBM or third party integrators. We additionally recommend that organizations "new" to BPM adoption work with an IBM or third party services organizations to both accelerate solution development and to ensure a successful design and deployment of initial BPM solutions.
So how do you architect BPM the "right way"? It is rather like our earlier point on what is a "good" BPM application. Everyone will have their own opinion on this, but we will discuss what our thoughts are based on numerous client engagements. Given the earlier article on design reviews and guidelines for BPM development, we will focus on three key areas: basic architecture, infrastructure and topology, and integration. This is where we have seen the key challenges as organizations develop BPM applications.
The lack of basic architectural considerations in BPM design and development is a paradox. It is like we have forgotten the basic tenants of application development. Many of our clients do not define and design for nonfunctional requirements (NFRs). The key NFRs that need to be considered (at a minimum) include performance, scalability, availability, security, portability, reliability, reusability, usability, extensibility, maintainability, legal/regulatory, and governance. In addition, some applications impose other key NFRs, but this is beyond the key points of our discussion.
We find that there are a number of often repeated anti-patterns in the area of basic architecture:
- Design of NFRs is secondary (or non-existent). Often nonfunctional requirements are not surfaced until after the application goes live and emails and telephone alerts indicate the failure of the development team to address the requirements in design.
- Data design both at the level of the BPM datastore as well as design of archival functions and interfaces to other external datastores. One customer assumed that they could just store the entire history of their process application instances for multiple years until they found that the database had grown in excess of 50GB and could not easily be queried.
- Performance is often not addressed until load test (or sometimes after the application goes "live"). We find that areas that are not well-defined include database performance planning and user interface issues in terms of rendering time, which directly maps to user dissatisfaction. We also see network bandwidth as a key consideration that is not well addressed by many of our clients.
- Security and entitlements are an afterthought. Many times, we find that the design of roles and entitlement management is neither defined nor developed until after the initial deployment, which causes extensive rework of the base BPM application.
- From a performance standpoint, we continually find that solution sizing is not done correctly. Much of this stems from the aforementioned lack of NFR definition, but additionally we find that business people tend to focus on average case versus worst case. One current client of ours did a great job of defining the key average load, but when the peak volume hit, they were unable to manage the performance load. Business users were frustrated and the application team had to rethink fundamental parts of their solution in terms of reducing contention and eliminating race conditions. It was not simply a case of "throwing" more hardware at the problem. More accurate performance definition up front would have circumvented many of these problems.
- We find that business users spend a great deal time defining the "happy path", but tend to do a minimal job of defining or thinking through exception cases. Often, these are discovered in user acceptance testing, and then require rework at the application layer to reconcile these issues. Though some may categorize these as functional requirements, the lack of a good approach for exception management almost always points to a lack in effective requirements definition and usually is evidenced as a systemic issue versus a use case issue. Additionally, we see that the implementation of a robust and consistent approach to exception handling is yet another issue in this area.
- When working with clients, we tend to spend a considerable amount of time of defining and testing availability as a central function of the application. We find that business users simply "expect" the application to run. Without a careful and rigorous definition of the availability aspects at each layer of the solution architecture, there is a good chance that an errant failure in one portion of the application will lead to an application that is unavailable and, at worst case, lead to transactional consistency issues. Though this becomes manifested at a technical level, it is a requirement that needs to be defined by both business and IT. It is much more than simply implementing the Gold BPM topology. We should note that the Gold topology is an absolute requirement for nearly all BPM applications and should be a prerequisite when determining the target topology for BPM applications. Without this focus, BPM will often get the blame when the solution fails to perform adequately.
There are a number of leading practices around basic architecture that we can quickly summarize:
- Define NFRs early and revisit throughout design.
- Spend additional time in early analysis and design to identify risks and mitigation approaches as well as addressing requirements and scope decisions.
- Adopt design and development techniques that support rapid refactoring and flexible modification based on the capabilities of the Process Center technology.
- Continue to invest in the creation and governance of design documentation (for example, use cases, system context, architecture diagram, operational architecture, exception path design, and architecture decisions) - BPM and agile development do not alter this need.
- Define and document each core NFR area and ensure business level support for the approaches taken to provide satisfaction of these areas.
- Define procedures around version management as well as software deployment best practices in terms of software release methodology and ensuring a consistent development and deployment release approach - this involves an understanding of the IBM Process Center capabilities.
Infrastructure and topology
We see a number of key anti-patterns in this area:
- Testing BPM is a critical function and we often find there is no defined end-to-end system testing topology and limited test cases. This is normally due to limited collaboration between stakeholders and development teams. A recent client engagement evidenced a lack of formal testing use cases or a documented test approach. The expectation was to go from user acceptance testing (which was not well-defined) into a limited production release. We talked the client into adoption of a more formal approach with a full production-sized topology to support acceptance testing.
- Though it enters into the larger area of integration, we find that there tends to be a lack of collaboration between the BPM development team and external system stakeholders. This problem manifests itself in security issues (such as not having appropriate credential policies defined), SLA issues (such as lack of understanding the impact of the BPM application on the current application or database operations), and transactional interactions (such as not understanding potential inconsistencies that might occur based on disrupted operations)
- There are a number of key network areas that are not well-defined in many BPM applications. The failure to consider network latency is a typical challenge we often see with BPM solutions. We had a client attempting to do two phase commits between a mainframe database and a distributed database. When we reviewed the implications of this on lock resources and potential for deadlock, it was clearly a problematic issue that necessitated both design and topology changes. We also find that issues of discontinuous connectivity are not well defined. This is particularly an issue in geographies with less than optimal communication quality. There are multiple ways to mitigate this both in the topology and the application design, but considering this issue early in design can alleviate trying to retrofit the solution in the future.
- As a last key observation, we see many clients attempting installation on a poorly sized configuration or trying to change the configuration after initial development. Though you can do some of this "on the fly", one current client using a single cluster, single node production configuration was rather shocked when we explained that they would need to "migrate" to a Gold topology to ensure an available solution. Learning by reading white papers is not a good approach for solution design regardless of product or technology, but we see this again and again when working with clients.
There are several key leading practices in this area:
- Establish a sound set of objective performance criteria and ensure they are fully tested prior to application release. As part of this point, make sure that system validation test environment is the same physical and logical topology as the production environment. We have seen many BPM applications deployed into a clustered environment only when it is released into production. Organizations need to fully test the application with a configuration that matches production. Some readers will argue that it is not feasible, practical, or cost justified having a non-production environment that is as big as production, and they may be right. Whether the non-production environment exactly replicates production, or whether the performance environment is scaled down physically (few cores or memory) is at the readers' discretion, but at a minimum, the topology used for performance testing should exactly match production.
- Organizations need to carefully consider and implement high availability (HA) and disaster recovery (DR) functionality and ensure that the functions are supported in the operational topology design. Do not "skimp" here as it will come back to "bite" you. There is a critical need for a reasoned discussion with business and IT to determine the HA and DR requirements and to properly classify the application in terms of HA and DR needs. This requirement additionally needs to focus on discussions and architectural decision in terms of defining the class of service that an application requires based on the business.
- From an operational standpoint, spend considerable time in the assessment and evaluation of shared services (such as network, security), external applications (both sources and targets), and external databases to ensure a consistent topology for development, test, and production. It is especially important to involve the owners of shared infrastructure services, such as LDAP and database servers, in the evaluation and planning of a BPM solution. Business processes that involve hundreds or thousands of users and millions of in-flight process instances can generate a heavy load on these shared services, as BPM relies on them for many of its underlying features and functions. We have seen numerous cases where a BPM application goes live and it proceeds to overwhelm the corporate LDAP servers. Who hasn't been frustrated at one time or another for having issues trying to access their email, or even their desktop because they can't login to the network? Don't be the application that brings down every other application in your company!
- Ensure that you build out the BPM topology to optimize the performance and availability via the clustered Process Center and Process Server runtimes to provide support for scalability, failover, high availability, and redundancy.
- Involve the database team to ensure that they are aware of the tuning considerations for BPM and spend the time to review and optimize the Process Server databases. As part of deployment planning, the team needs to document a plan for periodic maintenance over time such as runstats. (As a side note - there are a number of considerations specific for Oracle-based Process Server implementations that need to be reviewed and instituted.)
- If using IBM BPM Advanced, ensure the installation of IBM Integration Designer with the local Process Server UTE to support basic unit testing and encourage IT developers to use the UTE, not Process Center for unit testing.
As the last key "complex" area - integration continues to be a critical area where we see multiple challenges in client implementations. Normally, we view BPM design as a top-down design approach with business users developing key business level functional requirements. Unfortunately, we see too many clients focused nearly exclusively on this perspective and neglecting the key integration bottom-up design requirements. We see many clients that approach BPM from a strictly bottom-up SOA-oriented approach - and this is not a pattern for success either. But, we do find that these clients normally design and implement strong integration functions in their BPM applications, though unfortunately, they often miss the "forest for the trees" and usually do not do the collaborative design needed to support an optimal business-oriented design.
In the area of integration, perhaps the key principle to understand is that separation of concerns is fundamental to provide a layered approach to BPM. The use of key integration patterns (such as the facade pattern) is critical in the implementation of integration between the BPM layer and systems of record. As a related issue, we see multiple instances where organizations do not provide the basic transactional semantics in BPM applications.
One recent review of a client's BPM application evidenced a set of separate activities doing retrievals and updates to underlying transactional systems. When asked how they would deal with compensation and transactional semantics, the client gave us a blank stare thinking that the multiple activities would be dealt with as a single unit of work. We recommended using IBM BPM Advanced or using a transactional integration approach (such as IBM Integration Bus) or, at the very least, wrapping the multiple statements into a stored procedure to support atomicity and recovery. Each of these patterns and approaches needs to be reviewed as architectural decisions and the relative pros and cons need to be considered to develop an optimal approach.
We find a number of anti-patterns repeated across clients, and often these become difficult to re-implement if they are not well-designed in the initial implementation. These anti-patterns include the following:
- Limited (or no) consideration of transactional issues and potential race conditions leading to situations where the systems of record are left in an inconsistent state. The BPM Advanced solution provides for compensation, which may be a key requirement in transactional management.
- Lack of separation of concerns in the design of the integration with an ad hoc approach to integrating to enterprise systems. The creation of redundant integration services will result in application management issues.
- Lack of knowledge of SLAs for all components with a focus on external systems. This is critical in terms of satisfying key nonfunctional requirements for the BPM application.
- Limited (or no) use of existing service interfaces that leads to "reinventing the wheel".
- Integration implementation in Process Designer versus using the Integration Designer or ESB-based integration services. This is often an area where IBM and System Integrators can assist clients in understanding the relative advantages of a BPM Advanced approach for some types of transactional applications.
- Clients using the BPM Advanced solution often do not compartmentalize their integrations cleanly and end up creating numerous BPEL-based Advanced Integration Services. This approach often leads to the deployment of large numbers of EARs and JARs to Process Server, which causes management issues and can markedly increase the amount of time required to deploy BPM solutions.
Some of the leading practices to consider in the design of the integration aspects of a BPM application include:
- Organizations should use BPM Toolkits and Advanced Integration Services. Organizations should design solutions with a focus on factoring common services into toolkits – by employing a documented governance approach (based on reuse, application maintenance) so that the solutions can be managed more easily over time.
- In addition to the use of BPM Toolkits, organizations should employ the façade pattern to enable maintainability for integrations. We find that many clients hardcode endpoints, which causes high levels of churn in application maintenance.
- The use of standard ESBs enables an approach consistent with separation of concerns. This provides information hiding as well as supporting common mediation and transformation.
- It is important that a consistent snapshot policy should be developed when using Integration Designer and Advanced Integration Services. This should be done in tandem with an organization structure focused on a collaborative development process between BPM developers and integration developers as parallel approaches.
- Organizations need to be "prudent and practical" when choosing integration technologies and integration patterns. For example, organizations should avoid the implementation of request and reply integration patterns with message queuing, unless the integration mechanism supports asynchronous patterns and the extended latency is acceptable. It is important to evaluate each integration requirement individually and choose the right technology to fit the functional and non-functional requirements of the BPM application.
As a final point in this section, organizations developing BPM applications should adopt a formal approach for design reviews. Designing software is similar to creating a song or work of art - people do not like their creations criticized, but having objective and structured reviews are critical to the release of quality solutions. During the review, address the following areas:
- Begin the review by questioning and understanding the intent and objectives of the proposed solution.
- The review should drive the discussion from a position of fact and experience - it needs to be objective and measurable.
- Every project requires a review, but not every project needs a detailed review (it will overwhelm you and the team). Critical applications should have an assurance review that covers all of the dimensions. (As a sidenote - the referenced article by Scott Simmons contains a series of key considerations that can be used to develop a BPM design review approach.)
- There are different types of BPM reviews, depending on the solution implementation, such as BPM Standard (focused largely on BPM aspects) and BPM Advanced (with an additional focus on technical integration issues).
- Reviews may also differ by audience, such technical versus business, culture, and organization.
In short, there is no magic formula to doing a review. It is your responsibility to determine how to customize and align for your organization.
BPM application design and deployment is not a simple task - though some BPM solutions do not need the level of rigor we associate with other applications. We see too many BPM solutions being deployed that skipped the key steps in architectural design. It all starts with a clear definition of requirements and an understanding of the role of BPM as a solution approach. BPM can provide huge benefits if organizations perform the due diligence needed to ensure it is the right solution. Once BPM is selected, it is critical that organizations determine the underlying architecture needed to support the application in production. BPM applications, as a rule, need rigor to ensure that they will provide an effective and sustainable solution for the long-term. Finally, the use of design reviews to ensure that the BPM applications are designed in accordance with stakeholder requirements and take advantage of leading practices provides the essential checkpoint prior to releasing BPM solutions into production. By following these basic steps, organizations will find that BPM will deliver on the anticipated benefits. Conversely, the lack of careful design and architectural rigor will doom both the specific BPM application and will likely reduce the success of other BPM initiatives in the future.
Take the time to do BPM right - your business stakeholders will be grateful. Organizations that are just starting their journey with BPM are urged to work with IBM or third party integrators to ensure success with BPM solution development and implementation. Though they might not actually thank you directly - the proof will be in their adoption of the specific BPM application and, more importantly, BPM as a core solution competency for the future.
The authors would like to thank Bertrand Portier, Stuart Jones, Vinod Ralh, Paul Ilechko, and Dave Paskach for their critical review and additional insight into this area. The authors are fortunate to work with such a bright group of technical practitioners.
- IBM Redbook: IBM Business Process Manager Version 8.0 Production Topologies
- IBM Redbook: Combining Business Process Management and Enterprise Architecture for Better Business Outcomes
- IBM Redbook: Improve Design and Deployment of Processes Using IBM Business Process Manager V8.0
- IBM Redbook: IBM Business Process Manager V8.0 Performance Tuning and Best Practices
- Evaluating BPM applications: BPM design reviews and Rubik's Cubes
- Successful BPM takes a true team-oriented approach
- IBM Blueworks Live
- developerWorks BPM zone