Level: Introductory Bill Higgins (bhiggins@us.ibm.com), Architect, Systems Engineering and Architecture, IBM Global Services
15 Jul 2005 from The Rational Edge: Complex business process integration (BPI) projects can be challenging because different parts of the organization may follow different development processes and use different technologies and tools. This article suggests techniques for managing such projects along three fronts: organizational processes and tools, organizational structure, and requirements and change management processes.
 A key goal of modern enterprises is to integrate their business processes end-to-end across the company and with key partners, suppliers, and customers.1 Enterprise IT organizations pursue this goal by integrating the processes supporting software applications through business process integration (BPI) projects. Such projects can be challenging because different parts of the organization may follow different development processes and use different technologies and tools. Enterprises that find ways to optimize their BPI projects have a great market advantage: They can more quickly and effectively respond to emerging marketplace opportunities and competitive threats. This article discusses ground rules and techniques that your development organization can use to execute BPI projects more effectively.
First, we will examine the characteristics of a BPI project. Then, we will describe ground rules and techniques that fall into three categories:
- Organizational processes and tools
- Organizational structure
- Requirements and change management processes
We will discuss each category below.
 | |
Terminology
Throughout this article, I'll be using a number of terms that have wildly different meanings throughout the literature. The following definitions explain what they mean within the context of this article. If there is a similar term in the Unified Modeling Language (UML), I've enclosed that in parentheses.2
Business process integration project -- A business transformation project involving more than one business process area, more than one autonomous team, and many individual applications.
The supersystem -- The set of all applications involved in the development and testing of a business process integration project. (UML2: System)
Subsystem -- A set of related business applications treated as one unit within the supersystem. A subsystem is a development stakeholder that should be represented during supersystem scope determination, architecture, and change management activities. (UML2: Subsystem)
Application -- A unit of software functionality with a unique development team. In an integrated IT environment, a user may not be aware of separate applications involved in a particular transaction or workflow (since the click of a hyperlink may seamlessly transport the user across application boundaries). (UML2: Subsystem)
|
|
Characteristics of BPI projects
A business process re-engineering effort at a large enterprise is a complex undertaking that may take several years and involve many people and many IT systems. Typical BPI projects involve connecting, simplifying, and streamlining a large-scale business process and its supporting applications.
To understand the techniques described below for improving such a project's likelihood for success, it's important to first understand the differences between BPI projects and more typical application development projects.
Both BPI and application development projects deliver either new software components or new versions of existing software components that are intended to better support an enterprise's business processes. A software development team -- composed of project managers, analysts, developers, testers, and deployers -- work together to evolve a set of business goals into a deployable, dependable system that supports those goals.
Figure 1: A normal development project: one team, one system
However, BPI projects and application development projects differ fundamentally in terms of organizational and design complexity. A typical application development project might involve twenty to fifty people working on a system with 50,000 to 100,000 lines of code. A BPI project, in contrast, may involve between twenty and one hundred applications, each with its own team and sizable code base. The large number of applications is not the source of difficulty; rather, the complexity that arises from interdependencies between applications is the primary source of difficulty.
Increasing the number of interdependencies between applications leads to project diseconomies of scale. If one application depends on another, then the two application development teams must coordinate on development plans, test plans, and change management. Scale this out to between fifty and one hundred applications, and you wind up with a fragile web of interdependent applications and teams.
Figure 2: BPI project: many teams, many systems, and one integration team
Another challenge for BPI projects is that the application teams typically come from disparate, traditionally siloed organizations, so they tend to use different development processes, terminology, and tools. Though using slightly different tools or attaching slightly different meanings to key terms might seem like trivial problems, if you multiply this dissonance across many communication paths and many months of work, you get a great amount of needless friction between teams. So the next section will discuss the first ground rule for success on a business integration project: Use a standard process and toolset.
Standardizing organizational processes and tools
Before we discuss how to standardize your processes and tools, let's see why it is desirable to do so.
First, we should differentiate here between adopting any given methodology and adopting a standard methodology. A given team may realize productivity gains if it adopts any process at all instead of operating like an adhocracy. However, as different teams begin to work together on larger-scale business process integration projects, process differences between groups will lead to unnecessary friction and productivity losses.
Standardizing on a single process and toolset allows an organization to scale up to larger projects and achieve greater process integration by reducing complexity and eliminating a great deal of waste. The end results are reduced time-to-market, higher quality, and lower costs. These cost savings can be parlayed into additional user functionality, reallocated to other enterprise investments, or returned to shareholders.
Disparate processes and tools lead to increases in complexity and waste in a number of ways. When different teams use different processes, collaborating teams must struggle with each other's terminology, artifact and diagram structure, and ways of performing common tasks. For instance, one team might express functional requirements as use cases, whereas another might express them as traditional "shall" statements. These teams would have to struggle to express functionality that crosses system boundaries.
When projects use different tools, it is difficult, if not impossible, to share project data between teams, or to aggregate and analyze project data at higher levels of abstraction. On a BPI project, often one team needs to levy a requirement against another team to document a dependency. If these teams are using different requirements management tools, they will likely have to enter the same requirement in multiple systems, make parallel changes, and establish multiple traceability links to ensure consistency.
Unfortunately, establishing a standard process and toolset can be extremely difficult. Why?
- Teams are reluctant to move to a standard process and toolset unless it is their process and toolset.
- Immediate business needs and firefights often slow the progress of longer-term process and toolset standardization efforts.
- Team members require training on the new process and toolset before they can use it effectively.
Moving to a standard process and toolset requires strong executive support, a well-thought-out roadmap, and tenacious execution. Rather than trying to transition the entire enterprise within some arbitrary timeframe, I suggest that organizations begin using the new process and toolset within the context of a BPI project. These projects involve the integration of disparate process areas with many different teams, and they require effective inter-team collaboration to be successful. In such an environment, team members can easily see the enormous benefits of a standard process and toolset and recognize that they are requirements for success.
Focus on a few key team artifacts and related activities
A common mistake that organizations make is attempting to standardize everything in one fell swoop. This inevitably fails. Introducing too much change at one time results in a great amount of chaos, and team members retreat back to their old, fragmented processes.
A process is essentially a collection of roles, artifacts, and workflows. I suggest that organizations begin standardization efforts during a BPI project by focusing on key collaborative artifacts:
- Requirements
- Change requests
- Integrated project plan
- Technical interface specifications
Requirements and change requests determine the system's final scope. The integrated project plan ensures that dependencies between groups are managed and the project remains on schedule and within budget. Technical interface specifications define the contracts between the different groups that develop the applications.
There is no such thing as greenfield BPI development; you're always working with existing systems.3 In a sense, every requirement on a BPI project is also a change request. So within this context, we differentiate between requirements and change requests based on when they are accepted into the project scope. A requirement is a unit of scope you accept before baselining the iteration's scope; a change request is a delta to the scope (add, subtract, modify) after you baseline the iteration's scope.
Adopt new tools in an iterative manner
For some reason, normally sensible IT professionals often do not believe that the "laws of software physics" apply to the deployment of new software development tools. Although they understand the wisdom of developing business applications iteratively, they make the mistake of thinking that they can successfully deploy a whole new toolset at once.
A good rule of thumb is to adopt no more than one large-scale development tool at a time. Standardizing on such a tool is a long-term investment that incurs up-front costs, including training, data migration, and productivity hits as the team grows its expertise with the new tool.
A BPI project should focus on the new business capabilities it will produce, not on adopting new development tools. Introducing more than one major tool at a time can negatively impact the project's chances for success.
Retrain team members on process before deploying new tools
As we noted earlier, tools should align to your process, not define it. However, even if you customize a tool to align with your strategic development process, it will promote a certain "world view" that will inevitably influence your development process.
Attempting to deploy a new tool without retraining team members on the modified process can lead to negative situations:
- Team members resist using the new tool because they don't understand the process it supports.
- Team members attempt to shoehorn their old process into the new tool.
It's a worthwhile investment to train team members first on the soon-to-be standard process and later on how to use the supporting tools. This training can be informal, but it does need to take place in some form.
Deploy new tools during lulls in the development cycle
If you want your organization to move to a new tool, choose a time when team members have some flexibility. For example, if you want to introduce IBM® Rational RequisitePro® for managing requirements, a good time to begin the transition is near the end of the Elaboration phase, when the scope has stabilized and systems analysts will have more time for training. Another good time to train team members on Rational RequisitePro is in between projects: When developers finish their last Construction iteration on one project and are ready to move to an Inception phase for another project.
If you do decide to adopt a tool during a lull, be aware that project managers may resist funding the requisite training because they assume that many people will move on to other projects, so training them would be a waste of the project's money. To avoid this situation, it is best to fund training on tools and processes through professional development budgets provided by the employees' HR organization rather than through the project budget.
Now that we've looked at tools and processes, let's look at how organizational structure can influence the success of a BPI project.
Establishing a coherent organizational structure
As we discussed earlier, BPI projects are complex along many dimensions. They have multiple business process areas, multiple functional disciplines, disparate processes, terminology, and documentation styles. You can mitigate some of the effects of this complexity by establishing clear leadership and grouping the many applications into coarser-grained, more modular units. We will explore these techniques in the next couple of sections.
Establish one leader for each key project role
On a small team, it's relatively easy to establish leadership in key roles, such as project manager, architect, systems analyst, and change manager. However, large BPI projects often have leadership problems. Either there's no leadership at the project level for key roles, or more than one person tries to lead in each role and it's not clear who's in charge.
It is easy to understand why the first situation arises. Leading at the macro BPI level requires an individual with a great breadth of knowledge about the different processes and organizations involved as well as an ability to work with different groups who may follow different processes.
The second situation -- two or more individuals competing for a key leadership position -- commonly happens when two large organizations must collaborate on a larger-scale integration project. Leaders from the different organizations fight for turf at the BPI level. Though you might be tempted to create a joint leadership arrangement or leadership by committee, these are recipes for slow decision-making.
For the sake of timely and definitive decision-making, the project should identify a single leader in the following key project roles:
- Lead project manager
- Lead architect
- Lead systems analyst
- Test manager
These leaders should gather input from other team members, but ultimately, they must have the authority to make decisions that will optimally benefit the project.
What does it take to be a good BPI project leader? Ideally, individuals in the key roles should have a minimum of three years experience in a similar capacity across a broad swath of the organization's current application portfolio. These people will be acquainted with the most knowledgeable and influential people across the application portfolio, so they can quickly gather vital information, resolve potential problems, and delegate smaller-scale work items.
Group applications into smaller, semi-autonomous units
One large internal IBM project I worked on had upwards of a thousand applications, so we divided them into fifteen functional areas, organized around coarse-grained business processes, such as "market and sell" and "fulfill orders." Although each of these areas contained between thirty and four hundred applications, by working with senior project managers and architects who were familiar with the applications in each area, we were able to deal with only fifteen subsystems rather than a thousand applications.
It's simply not possible for one person to understand the details of more than one or two complex applications. A large BPI project may involve tens or even hundreds of applications. If this is the case, consider modularizing the project structure by grouping the applications by business process area. For example, you might group all financial applications or all Web commerce applications together.
Figure 3: Non-modular view versus a modular view of applications
Then, nominate one lead project manager and one lead architect from each application area to represent that area at the project level. This allows technical management to consider the project's complex supersystem at a coarser level of granularity.
Establish representative committees for each key functional discipline
In my role as a business process / integration architect on another large BPI project, I helped run a weekly architectural coordination call. Although we had about ten big subsystems in the supersystem, representatives from only seven of these subsystems routinely showed up for the weekly architecture checkpoint.
The calls themselves were frankly unpleasant; most of the representatives were very opinionated, and ugly clashes of egos and technical opinions were common. However, life eventually became much more unpleasant for those who did not attend these early architectural calls. As we finished our first development iteration and went into integration test, those subsystems not represented on the calls had the biggest integration problems.
The fact is that early meetings may be unpleasant simply because you are dealing with problems earlier rather than later. Teams that skipped the meetings and arguments weren't making the underlying problems go away; they were simply deferring them until later.
The best way to reduce project risks up front is for each key functional discipline (project management, requirements, architecture, test, change management, and deployment) group to hold frequent coordination calls with their subsystem leaders. In addition, subsystem leaders should form a "change board" that meets regularly to review and approve any changes that cross subsystem boundaries.
Figure 4: Functional leadership and coordination
Now that we've talked about organizing the project for success, let's talk about a couple of specific, crucial development processes: requirements and change management.
Optimizing the requirements and change management processes
As we discussed earlier, requirements and change requests, respectively, are two of the most important artifacts on a BPI project. Below are specific techniques for optimizing these project elements.
Use an iterative methodology to deliver functionality
I strongly recommend that you follow an iterative deployment process for your project that allows you to go from concept to running system in a matter of weeks -- not months or even years.
This is especially crucial for a BPI because the new functionality is often discussed at such a high level of abstraction, allowing for widely different interpretations by different stakeholder groups. Delivering most of the new functionality early, over a couple of iterations, allows stakeholders to see, touch, and react to it, so you can quickly drive out ambiguities, redefine requirements if necessary, and discover and solve problems when they're easiest to fix.
Another reason to use an iterative approach is that the subsystems for integration in a BPI project are actually large applications or collections of large applications. The traditional waterfall approach delays subsystem integration until very late in the development lifecycle, when you are likely to experience a high defect rate and find yourself with a great deal of rework.
If you use an iterative approach, ideally, your test environment should mirror your production environment; at a minimum, it should simulate that environment. The cost of doing this might seem prohibitive for a BPI that includes many applications and potentially hundreds of servers. But if you consider the cost of massive rework on a project of this scale versus an up-front investment in a large test environment, I'm confident that you'll choose the latter.
One caveat: Iterative development is easier if you're simply adding new functions to an existing architecture and harder if you're making major architectural changes. If you're making a fundamental architectural change to the supersystem -- for example, replacing a homegrown ERP system with an SAP deployment -- you may be tempted to take a big bang approach. But you can use a more sensible iterative approach instead, even if you're undertaking major architectural changes. Using the SAP example, you could transfer functionality from your custom applications to SAP over a number of releases, gradually turning on more and more native SAP functions.
Allow only one representative from each stakeholder group to submit requirements and change requests
On large projects, hundreds of workers may be directly or indirectly involved with project execution. Things can quickly get out of control if anyone associated with the project is able to submit new requirements and/or change requests. Instead, the best practice is to allow each stakeholder group to nominate exactly one representative who will be authorized to submit requirements and change requests on behalf of the group.
What constitutes a stakeholder group? Ideally, the group should consist of the same set of stakeholders you identified when you established the project's vision during the Inception phase. This might include business sponsors, future system users, and the development team.
It is very common for a large BPI project to have multiple business sponsors. In a typical supply chain integration project, for example, marketing, sales, fulfillment, and finance are all stakeholders. Each of these groups should have exactly one representative to submit requirements and change requests.
Future users include customers and business partners who interact directly with your company via the system (e.g., a Web or phone interface) and employees who will use the system to support their work. These users are typically represented by company proxies (such as a customer support organization), since it would be impossible to manage direct interaction with hundreds or thousands of actual customers, business partners, and employees.
The development team on a BPI project is actually a team of teams, each responsible for a large subsystem of the supersystem. Since a large BPI project may involve tens or even hundreds of applications, these subsystems should actually be collections of applications related to a business process area, as we described earlier. Then, a representative of the development team for each subsystem should be allowed to submit system requirements to the change board that state a dependency against another application and propose change requests that modify these requirements later in the development cycle.
Differentiate among change requests
To respond intelligently to a change request, you must first understand the reasons for it. Though these are nearly infinite, only a few legitimately justify system alterations:
- A change in the external business environment
- Flawed understanding of the business problem
- A technical flaw in the solution
A change in the external business environment. Such changes can have a broad range of impacts, from dictating a trivial scope adjustment to rendering the entire project obsolete. For instance, in 2004, IBM had many projects in progress to improve its personal computing (PC) business. The company's announcement that it was selling this business to Lenovo stopped many of these projects dead in their tracks, as the business focus shifted to divestiture efforts.
Generally speaking, the business stakeholders who chartered and funded the project should announce changes in the external business environment to the project team. Ultimately, these are the folks who make decisions about altering project scope; they work with the lead architect and lead project manager to determine how to respond to the change. Assuming the project survives, BPI-level project managers work with managers for individual subsystems to make subsequent planning and design changes that accommodate the business change.
Flawed understanding of the business problem. That the development team will have an imperfect understanding of the business problem at the outset of a project is almost inevitable. However, as the project progresses from concept to prototype to running system, the business and development teams' understanding of the business problem should evolve from fuzzy to concrete. Skilled development teams use techniques to achieve more clarity early on, including prototyping and shorter iterations. When they discover problems stemming from misunderstanding of the business problem, it's up to the lead architect to work with business stakeholders to update the design, and then the changes will affect one or more subsystems.
A technical flaw in the solution. A technical flaw is different from other legitimate change-drivers because it is wholly unrelated to the business domain; typically, it is brought forward by someone on the development team rather than someone on the business team. Technical flaws are especially prevalent and problematic at subsystem boundaries. One team might realize that its subsystem needs an extra data element from an upstream subsystem in order to behave correctly. These changes should be brought before the change board by the subsystem representative. The only exception to this rule is if a design flaw is completely isolated within a single subsystem, in which case the subsystem team can manage the change on its own. We will examine this situation in more detail below.
Process local changes within subsystem teams
A downside of having a centralized, higher-level BPI project leadership team is that it is easy for this group to become a bottleneck as you move into more detailed work during the project's Elaboration and Construction phases. If every requirement or change request, no matter how small, must pass through a change review board, for example, progress will either slow to a crawl or people will find creative ways to bypass the system. Instead of trying to control every aspect of specification (requirements and design), it is better for the leadership group to actively push specification activities into lower project levels.
Changes stemming from events in the external business environment (see previous section) should go through the change board, because the lead architect must determine how the change should be implemented across the supersystem. However, changes that stem from discovering a design flaw may or may not require the change board's attention.
Fixes that require changes to two or more subsystems should go through the board to ensure that all parties agree to the solution. However, if the requirement or change request is local to one subsystem, then the subsystem team can handle it alone. They are the experts in that area and can deal with the change quickly and definitively; reviewing it with the change board would not add value and would only slow down resolution. You should involve the change board in a local work item only if the change might significantly affect the subsystem's schedule and therefore the supersystem's schedule as well.
Avoid temporary, manual workarounds
BPI project teams working on tight schedules may feel they need to resort to a manual workaround in order to avoid a complex software design point. There are two types of manual workarounds: one-time only changes and ongoing process steps.
One project I worked on pulled off a one-time workaround quite successfully. A main requirement of this project was to move from a custom CRM application to a vendor's CRM package. This involved customizing the packaged CRM application to match existing capabilities and also migrating custom data sources to the CRM's database. Midway through the Construction phase, the developers discovered that they had overlooked the need to migrate a small, custom database that included a relatively modest but important portion of the overall business data. Writing scripts for the migration would break the schedule, they realized. Everyone panicked. However, eventually someone suggested that, since the database was small, instead of writing complex scripts, perhaps they could hire an army of temp workers to manually enter the data into the new system over the course of a week. This solution worked. The data was migrated for a relatively low cost, and without breaking the project's schedule or budget.
One-time workarounds like this one may be preferable to developing software that you'll only use once (unless that software operates on a large dataset).
Less appealing are "temporary" manual workarounds that will become part of daily operations. The cost of performing such a workaround is often quite high (you must hire people to perform the work), and then it may take much longer than you anticipate to replace it with a permanent IT solution -- if it is ever replaced at all. If, as a business owner, you are ever asked to approve a temporary manual workaround, make sure you get approval for a requirement calling for a permanent IT solution in a future release -- one that has a low risk of being cancelled.
Conclusion
In this article, I've discussed some of the unique characteristics and problems of BPI projects. I then described some ground rules and techniques you can use to deal with these problems. Going forward, we're likely to see more and more large-scale integration projects, so it's important to learn how to confront these problems head-on and resolve them efficiently. Teams that can accomplish this will create more useful systems and give their organizations a sharp competitive edge in the marketplace.
Acknowledgements
Thanks to the following individuals for teaching me many of the ideas I incorporated into this article: Carolyn Maher, Grady Booch, Jay Strosnider, John Hartley, James Jamison, Kurt Bittner, Ralph Nelson, Russ Taylor, and Simon Johnston.
Thanks also to Marlene Ellin for her patient advice during the editorial process and to Kurt Bittner and Michael Hanford for providing feedback on early iterations of this article.
Further reading and references
Kurt Bittner and Ian Spence, Use Case Modeling. Addison-Wesley Professional, 2002.
Frederick P. Brooks, The Mythical Man-Month: Essays on Software Engineering, 20th Anniversary Edition. Addison-Wesley Professional, 1995.
Robert Galen, Software Endgames: Eliminating Defects, Controlling Change, and the Countdown to On-time Delivery. Dorset House, 2004.
"IBM's Sam Palmisano's Definition of On Demand Business" at http://www-5.ibm.com/e-business/za/about_ondemand/def.html
James Rumbaugh, Ivar Jacobson, and Grady Booch, The Unified Modeling Language Reference Manual, Second Edition. Addison-Wesley, 2004.
Notes
1 See "IBM's Sam Palmisano's Definition of On Demand Business" at http://www-5.ibm.com/e-business/za/about_ondemand/def.html
2 For complete definitions of the referenced UML terms, see James Rumbaugh, Ivar Jacobson, and Grady Booch, The Unified Modeling Language Reference Manual, Second Edition. Addison-Wesley Professional, 2004.
3 Even a "new" application is not likely to introduce brand new functionality into the supersystem.
4 Kurt Bittner and Ian Spence, Use Case Modeling. Addison-Wesley Professional, 2002.
About the author | 
| | Bill Higgins, an architect with IBM Global Services, works on collaborative development technologies with IBM's On Demand Workplace and Rational organizations. Currently, he is researching portal-based solutions to assist software development teams. His technical interests include the IBM® Rational Team Unifying Platform,® IBM Lotus Workplace,® IBM WebSphere Portal Server,® mapping business processes to IT, and recording his activities and insights in his IBM developerWorks blog. He holds a BS in computer science from Penn State University. |
Rate this page
| 
