Over the past 10 years, the software community has seen significant adoption of agile practices. These practices were a reaction to the deficiencies in the existing waterfall-style process for software development:
- Slow delivery
- A waterfall approach creates executable (and reviewable) system over many months, or even years, thereby reducing stakeholder opportunities for feedback and limiting business flexibility.
- Deciding early
- With limited opportunities for review and input, stakeholders must decide very early on features that are important to the system's success.
- Limited opportunities to adapt
- Long-term, fixed plans reduce the ability to adapt to new circumstances, both from technical discoveries as well as changes in the business.
In contrast, agile practices continuously integrate small system changes and provide an environment for continuous review. They break large projects into relatively small user stories and tasks and then integrate them into a continuous integration and build process. Continuously integrating changes reveals errors early and provides for early system verification. It also enables more frequent review, which enables early validation.
In a typical scrum process, stakeholders (product owner) prioritize a list of work (user stories) in the product backlog. At the start of each iteration, the product owner selects the highest-priority work items from the product backlog and discusses their details with the development team. The development team then breaks those stories into tasks to complete over the upcoming two- to four-week sprint (iteration). During the sprint, the development team meets continuously (daily scrum meetings) and continuously integrates changes. At the end of the sprint, the development team demonstrates the new capabilities to the product owner from the latest system build.
Many of the early agile success stories shared common characteristics: small teams, co-located, building relatively simple IT-type web or database applications. However, complex systems development can and do benefit from these agile values, including faster delivery, deciding late, continuous integration, early feedback, and adaptive planning.
For examples in this article, a hypothetical telecommunications company provides a product line of embedded parts. The article describes the agile practices that they follow, the challenges they faced in their existing development infrastructure, and how they addressed those challenges when moving to IBM® Rational Team Concert™ collaborative software.
A telecommunication supplier for the cellular industry provides parts for cell phones. Their teams are organized into a hardware team, firmware (software) team, testing team, and an app support team that creates development tools and test benches, and supports customers that have issues. Customer product leads (CPLs) manage relationships with clients and bring enhancements requests and defect reports back from customers to the engineering teams. The CPLs are similar to the product owner role in scrum project management.
Previously, the CPLs used a spreadsheet to track work. Development was continuously prioritized by moving rows up or down within the spreadsheet. The engineering team used a separate system, developed in-house, to track defects found by tests and by customers. The firmware team had two backlogs of work: the spreadsheet and the issue-tracking defects. The engineering manager collected metrics by writing Visual Basic (VB) macros in Microsoft Excel to collect data from outside of the issue tracking system. They handled configuration management (CM) with an existing, branch-based tool.
Figure 1. Project's organizational structure
The economic downturn of the late 2000s represented an opportunity for the business to expand its customer base. Each customer had unique requirements and possible additional hardware needs. What began as a single product on one hardware platform grew to 18+ product variants across four hardware platforms. Figure 2 depicts part of that growth from a single Product A to variants across multiple hardware platforms. Each stream represents a product variant, and the arrows represent how changes might flow between variants.
Figure 2. Product's software and hardware variant flow
The cellular communications industry is highly dynamic and agile. Manufactures are continuously integrating new versions of their suppliers' parts for feedback, to reveal issues early and show progress. Therefore, suppliers are required to frequently deliver new versions, sometimes multiple times a month, in response to that feedback. The CPLs are in constant communication with manufactures to decide what changes to include in the upcoming delivery. Based on those conversations, the CPLs re-plan work with the firmware team lead twice a week (twice-weekly sprints from an agile perspective).
Expanding customers led to large challenges for the firmware team for several reasons. Each new customer typically has custom enhancements. Not all changes (enhancements and defect fixes) are delivered to every customer variant because of factors such as cost of the feature, hardware platform, available memory, and so forth. To make matter worse, customers might choose to take changes in a different order. In the example illustrated in Figure 3, Customer A requires a change be delivered without the two preceding changes. The preceding changes might or might not be delivered to Customer A's variant sometime in the future.
Figure 3. Changes delivered out of order to Customer A
The expanding customer base also challenged engineering to meet the selected variation and schedules of new customers. Managing change in this environment is complex, because not all changes will be delivered to all product variants, or they might be delivered in a different order. To track the work, each customer variant has its own backlog of prioritized work, as shown in Figure 4. Note that there is a single development team, the firmware team, that implements all of these backlogs.
Figure 4. Tracking work across product variants
As new customers were added, their work backlog was recorded in a new tab in the Excel spreadsheet. Rows representing work (defects or enhancements in the form of user stories) were copied between tabs to record the fact that a particular customer was interested in a particular change. However, in a spreadsheet, there is no way to relate duplicate work as Figure 5 shows. There was no indication, for a given change, which product variant had the change and when it would be delivered.
The expanding customer base also challenged their legacy CM system. In branch-based CM systems, each file modified has to be placed on a separate branch, as shown in Figure 5. A change might touch 5, 10, or more than 100 files, and that requires the firmware engineer to create the proper branch for every file.
Figure 5. Existing change management approach to integrating product variants
Most merges in this environment are complex, because not all changes will be delivered to a particular customer or (worse) may be delivered out of order. Complex merges in branch-based CM systems are error-prone, because the CM systems do not manage a change; they manage only individual file variants. The engineer performing the merge is responsible for understanding which files are associated with the change, which branch those files are on, and where they need to merge. That's a time-consuming, error-prone process.
Merge errors are costly because they are often difficult to locate. Because of this complexity and potential cost, the firmware team had established a role called the Integrator. Only Integrators were allowed to merge changes to customer branches. Unfortunately, Integrators became a bottleneck in the process and sometimes caused delays in getting features or defect fixes into the releases that needed to be delivered multiple times per month.
Although not explicitly calling themselves "agile," the companies in the cellular communications industry follow many agile principles, as this one does. They frequently integrate changes (continuous integration), even across suppliers, multiple times per month. And based on what they learn from frequent conversations with the customer, they continually reprioritize their product backlogs (adaptive planning). For this organization, that happened twice weekly.
Agile practices of smaller work, adaptive planning, and continuous integration can overwhelm a development organization not prepared to handle these changes. Their existing tools were not designed to handle the frequent change and integration that we see today. In many cases, we now see that the organization's change process takes more time than the change within the system. The overhead of this change process is often the fault of the change management tools, as it was in this case: outdated CM system, Excel, VB macros.
In summary, this engineering organization faced three key challenges:
- Excel was insufficient for planning and tracking work across product lines. The process quickly fell apart when they tried to manage and prioritize changes across multiple product lines and then tried to understand which changes had been made to which variants and when a particular change would be made to a variant.
- The branch-based CM software was inadequate to meet their agile, continuous integration needs. There was no simple way to relate code changes to the work tasks in the spreadsheet plans, so it was difficult to see what changes others had made in previous product variants. The CM tool did not help with complex merges, which led to the need for an Integrator role to perform the merges. Integrators became a bottleneck and delayed delivery of changes.
- The engineering manager wasted time creating VB macros in an Excel spreadsheet to connect to the development repository for status information
The team chose to adopt Rational Team Concert to address those challenges. The organization had three primary needs:
- A more powerful configuration management tool
They needed a solution that helped more easily identify all of the file variants to be merged when they were selecting a change into a product variant. It had to provide an easier way to integrate changes so there was no more need for the Integrator role. The software also had to support relationships between the Excel spreadsheet tasks and the code changes being made, so that they could better understand what code changes were made when reapplying that change in a different product variant.
- Simplified planning across product lines
They needed to create separate but related work tasks for a common change that would apply across multiple product variants, so they could be independently managed and prioritized. The work tasks needed to be related so that CPLs could tell which product lines had that defect or enhancement, and so development could locate all of the code changes for that defect or enhancement. And they needed a mechanism that would simplify their work scheduling within and across product lines.
- Easier and better real-time visibility of development status
The engineering manager was tired of updating and running VB macros to get the status of the development process.
This section describes how the team used Rational Team Concert to address their challenges relating to configuration management.
Relating code changes to work
To address the first need, Rational Team Concert provides automatic relationships with work items. Software developers can easily associate a collection of code changes with the work task assigned to them. The screen segment Figure 6 shows the work items and their associated file changes. This enables developers to quickly find out what changes were made on another product line without looking at the code branches. Before Rational Team Concert, engineers had to look at the code branches to find what someone else did on another product line. Now they simple locate the work task and see all associated changes sets.
In addition, developers can use Rational Team Concert to work on multiple changes simultaneously in the same workspace. They do not need to create multiple local copies of the source code repository and remember which local copy is associated with which change. The developer simply selects the work item to suspend or resume work, and Rational Team Concert automatically configures the local sandbox with those changes.
Figure 6. Working multiple changes in Rational Team Concert
Stream-based configuration management
Rational Team Concert stream-based configuration management provides a simpler, more powerful approach to managing product line changes. Figure 5 showed the challenges of selecting an individual change and then finding and merging all related files. Rational Team Concert automatically identifies all file changes that must be merged into the incoming stream (product variant).
Figure 7. Integrating changes into products lines using Rational Team Concert Streams
One limitation is that Rational Team Concert uses a patch solution when change sets are delivered out of order. The challenge with the patch mechanism is that we lose traceability to a common change (however, you will see a solution to this limitation later in this article). The Rational Team Concert team understands that they need a better solution, and there are several work items in the Rational Team Concert backlog to address this issue. For more information, see the Rational Team Concert work item number 128329.
Help with complex merges
Rational Team Concert helps address complex merges in multiple ways. First, it provides a view of all the factors necessary to merge a change into a new stream (product variant). Therefore, there is no mistaking which files need to be merged. As Figure 8 shows, Rational Team Concert lists the files for each change set and indicates where there is a potential conflict in the merge.
Figure 8. Rational Team Concert merge
Developers can also take snapshots of their personal workspaces before a complex merge. During the merge process, Rational Team Concert automatically stores all merge-related changes in a separate change set. If the merge is not going well, the developer can revert back to the snapshot state by either suspending or discarding the change set.
Figure 9. Workspace snapshot
Multiple, linked tasks for work across product lines
Developers can duplicate a work item to create an independent copy with a reference to the original. Duplicate work items are useful for the firmware team to create the work performed across multiple product lines. The independent (but linked) work item can be separately planned and developed. You can quickly view the links on the original work item to see what other variants will have this change and whether they have been delivered yet.
In Figure 10, Story 174 is created as a duplicate of Story 57. When the developer begins work on Story 174, he can follow the link to the related Story 57 (highlighted in Story 174) and then accept Story 57's change set (highlighted in Story 57 and shown as the red change set) into his workspace. Accepting the previous story's change set automatically places all of the file difference into the developer's workspace. New changes required for the merge are automatically placed in a new change set and associated with Story 174, as shown in the figure.
Figure 10. Linking work across product variants in Rational Team Concert
The firmware team also uses Rational Team Concert process controls. Broken builds dramatically slow progress for agile teams that use continuous integration practices. Rational Team Concert enables teams to lock down delivery to streams so that only work that is currently planned for release to a customer can be delivered. Although someone might think that any working software could be delivered during an iteration, embedded software is sensitive to time and space issues, and software not planned for a current iteration can distract from the upcoming release and break builds.
However, the firmware team also wanted to enable engineers to collaborate and explore implementation solutions for future work. The solution was to enforce a rigid delivery policy for the customer streams and then create a Rational Team Concert sandbox team with more relaxed delivery rules. Developers who were members of the sandbox team could create additional streams for collaboration, using the less restrictive rules.
Spreadsheet planning has many limitations. It is difficult to manage and track the duplicate work across product lines, reprioritizing is hard, and there is a general reluctance to create detailed work tasks because creating and moving them is so difficult. As a result, many work tasks associated with a change (updates to the hardware test bench, modification to development tools) are not tracked and can be lost.
Child work items
Rational Team Concert was one of the first issue-tracking systems to simplify relationships between work items. The software makes it simple to create relationships and then later navigate and query on them. Embedded systems development might require downstream tasks after firmware changes are made — updates to the test bench, customer development tools, user docs. You don't want to lose those tasks, of course.
Those tasks would not appear in the spreadsheet planning, so they were at risk of being lost. Rational Team Concert relations between work items allows engineers to define this additional work and track it to ensure that it gets completed before delivery.
Planning across product lines
Planning across multiple product lines had long outgrown the spreadsheet. The CPLs and firmware team needed a mechanism to show what work was planned for which iteration and to easily allow work to move between iterations.
Rational Team Concert agile planning capabilities solved many of these issues. Work can be viewed in many customized plan modes, enabling the CPLs (product owners) to view work for a single product line. The week after the new software was installed, the CPLs and firmware team leads used Rational Team Concert planning tools, rather than the spreadsheet, to drive the twice-weekly planning meetings, with great success.
Figure 11. Contrast spreadsheet and Rational Team Concert planning
Although it was successful, there are some challenges in using Rational Team Concert for planning across multiple timelines. The planning is based on agile principles. Timelines are decomposed into nested iterations (release-sprint) and, at any level, only one iteration is current (Release 2.1, Sprint 3). Agile teams work against a single timeline. To support these agile practices, the plans select a single team (or team of teams) and single iteration (release, sprint) for planning and tracking work. In product line systems, a single team is working multiple product lines, each with their own schedules.
There is a problem here where a single team (firmware) serves multiple backlogs, one for each customer variant. To see work for different customers, the firmware team needs to change the plan's Planned For property constantly. An agile advocate would argue that the CPLs should internally prioritize their work across customers and present a single, unified backlog to the firmware team. However, that is not a reasonable request, and there are Rational Team Concert enhancement requests to select multiple timelines on a single plan (for more information, see the Rational Team Concert work item 94056 ).
This organization chose to have the CPLs track the work in separate timelines. When the agile teams want to see their work, they change which timeline their plan references. Although not an ideal solution, changing the plan timeline was a good workaround, and the overall solution is significantly better than the spreadsheet.
To report on engineering status and execution, the engineering manager wrote Visual Basic scripts in Excel to pull data from their previous issue-tracking system. Rational Team Concert work item queries and included dashboards provided a zero-effort solution for tracking progress. The engineering manager started configuring his dashboard the first week that Rational Team Concert was deployed.
Figure 12. Project visibility in Rational Team Concert
This article shows how a telecommunications supplier developing embedded products across multiple product lines is applying agile practices. As business grew their customer base, the firmware team was challenged to meet delivery schedules due to a combination of factors:
- Excel was insufficient for planning and tracking work across product lines. The process quickly fell apart when they tried to manage and prioritize changes across multiple product lines and then to understand which changes had been made to which variants and when a particular change would be made to a variant.
- The branch-based CM software was inadequate to meet the agile, continuous integration needs. There was no simple way to relate code changes to the work tasks in the spreadsheet plans, so it was difficult to see what changes others had made in previous product variants for a particular change. The CM tool did not help with the complex merges, which led to the need for an Integrator role to perform the merges. Integrators became a bottleneck and delayed delivery of changes.
- The engineering manager wasted time creating VB macros in an Excel spreadsheet to connect to the development repository for status updates.
Rational Team Concert resolved almost all of these issues with the changes shown in the table below.
Table 1. Changes resolved by Rational Team Concert
|Planning||Microsoft Excel spreadsheets||Rational Team Concert agile plans|
|Tracking duplicate work across product lines||Try to locate across multiple tabs in the spreadsheet||Link the related work items|
|Managing code for product variants||Branches in the SCM tool||Rational Team Concert stream per variant|
|Locating work performed in previous product version||Find appropriate branch in the SCM tool||View change set associated with Rational Team Concert work items|
|Who integrates code||Special "Integrator"||Everyone|
|Performing complex merges||Error-prone, branch-based merge process||Deliver change sets to customer stream, and Rational Team Concert identifies the required files to merge|
|Additional work for test bench, customer development, tools||Firmware team "remembers"||Rational Team Concert child work items|
|Status||VB macros in Excel||Rational Team Concert dashboards and plans|
It is interesting to note that when first talking with this customer, branch-based integration was the issue they discussed. But they quickly realized that the Rational Team Concert work item and planning capabilities would be a much better solution for the product-line planning and tracking.
- Learn more about agile development practices from these webcasts:
- Build quality products by leveraging agile development and automated metric reporting (IEEE, 2013)
- Continuous Integration for Agile Embedded Software Development (IEEE, 2012)
- Agile Development in a Software and Firmware Co-Development Environment (Dr. Dobb's)
- Read Continuous integration for agile development: How agile methods, continuous integration, and test-driven enhance design and development of complex systems, an article by Martin R. Bakal, Jennifer Althouse, and Paridhi Verma (IBM® developerWorks®, August 2012)
- Find out more about Rational Team Concert:
- Find Rational Team Concert articles and links to many other resources on IBM developerWorks, and check the product overview page, features and benefits, system requirements, and the user information center.
- Check the Rational Team Concert page on Jazz.net.
- Watch the Using Rational Team Concert in a globally distributed team webcast or a demonstration of the Dashboards and reports, or listen to the podcast about IBM Rational Team Concert and Jazz.
- Explore the Rational software area on developerWorks for technical resources, best practices, and information about Rational collaborative and integrated solutions for software and systems delivery.
- Stay current with developerWorks technical events and webcasts focused on a variety of IBM products and IT industry topics.
- Attend a free developerWorks Live! briefing to get up-to-speed quickly on IBM products and tools, as well as IT industry trends.
- Watch developerWorks on-demand demos, ranging from product installation and setup demos for beginners to advanced functionality for experienced developers.
- Improve your skills. Check the Rational training and certification catalog, which includes many types of courses on a wide range of topics. You can take some of them anywhere, anytime, and many of the Getting Started ones are free.
Get products and technologies
- Download Rational Team Concert from Jazz.net and try it free on up to 10 developers for as long as you want (requires registration). If you'd prefer, you can try it in the sandbox instead, without installing it on your own system.
- Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.
- Get connected with your peers and keep up on the latest information in the Rational community.
- Rate or review Rational software. It's quick and easy.
- Share your knowledge and help others who use Rational software by writing a developerWorks article. Find out what makes a good developerWorks article and how to proceed.
- Follow Rational software on Facebook, Twitter (@ibmrational), and YouTube, and add your comments and requests.
- Ask and answer questions and increase your expertise when you get involved in the Rational forums, cafés, and wikis.
Harry Koehnemann began his career at Intel Corporation, developing and supporting operating systems and compilers. He spent most of the 1990s consulting and training at large for software-centric systems organizations and the .com bubble at a still-successful Internet startup (employee #4), developing Java Enterprise systems using agile practices. After the bubble burst, Harry returned to consulting and has helped countless Fortune 500 companies in A&D, automotive, electronics, medical, and financial industries with their engineering and software lifecycle practices. He is currently Director of Technology at 321 Gang, an IBM Rational Premier Business Partner, where he continues pursuing his passion for helping organizations be successful in large, complex systems development. Harry also taught for more than 10 years as an adjunct professor at Arizona State University in the areas distributed computing, systems modeling, and embedded software.