Being agile in embedded product-line development with Rational Team Concert

Agile success stories are commonplace in the IT domain, but little has been published about agility for embedded, product line systems. Harry Koehnemann delves into the details of one embedded product organization’s adoption of IBM® Rational Team Concert™ to support their agile engineering practices. Harry Koehnemann of 321 Gang explains how their hardware, software, and project management teams collaborate, their use of agile techniques, and their migration to Rational Team Concert. Find out what problems they faced, the practices and tool changes that helped them, and what challenges remain.

Harry E. Koehnemann (harry@321gang.com), Director of Technology, 321 Gang

Harry Koehnemann began his career at Intel Corporation, developing and supporting operating systems and compilers. He spent most of the 1990s consulting and training at large for software-centric systems organizations and the .com bubble at a still-successful Internet startup (employee #4), developing Java Enterprise systems using agile practices. After the bubble burst, Harry returned to consulting and has helped countless Fortune 500 companies in A&D, automotive, electronics, medical, and financial industries with their engineering and software lifecycle practices. He is currently Director of Technology at 321 Gang, an IBM Rational Premier Business Partner, where he continues pursuing his passion for helping organizations be successful in large, complex systems development. Harry also taught for more than 10 years as an adjunct professor at Arizona State University in the areas distributed computing, systems modeling, and embedded software.



30 April 2013

Also available in Chinese Japanese

Overview

Over the past 10 years, the software community has seen significant adoption of agile practices. These practices were a reaction to the deficiencies in the existing waterfall-style process for software development:

Slow delivery
A waterfall approach creates executable (and reviewable) system over many months, or even years, thereby reducing stakeholder opportunities for feedback and limiting business flexibility.
 
Deciding early
With limited opportunities for review and input, stakeholders must decide very early on features that are important to the system's success.
 
Limited opportunities to adapt
Long-term, fixed plans reduce the ability to adapt to new circumstances, both from technical discoveries as well as changes in the business.

In contrast, agile practices continuously integrate small system changes and provide an environment for continuous review. They break large projects into relatively small user stories and tasks and then integrate them into a continuous integration and build process. Continuously integrating changes reveals errors early and provides for early system verification. It also enables more frequent review, which enables early validation.

In a typical scrum process, stakeholders (product owner) prioritize a list of work (user stories) in the product backlog. At the start of each iteration, the product owner selects the highest-priority work items from the product backlog and discusses their details with the development team. The development team then breaks those stories into tasks to complete over the upcoming two- to four-week sprint (iteration). During the sprint, the development team meets continuously (daily scrum meetings) and continuously integrates changes. At the end of the sprint, the development team demonstrates the new capabilities to the product owner from the latest system build.

Many of the early agile success stories shared common characteristics: small teams, co-located, building relatively simple IT-type web or database applications. However, complex systems development can and do benefit from these agile values, including faster delivery, deciding late, continuous integration, early feedback, and adaptive planning.

For examples in this article, a hypothetical telecommunications company provides a product line of embedded parts. The article describes the agile practices that they follow, the challenges they faced in their existing development infrastructure, and how they addressed those challenges when moving to IBM® Rational Team Concert™ collaborative software.


Project background

A telecommunication supplier for the cellular industry provides parts for cell phones. Their teams are organized into a hardware team, firmware (software) team, testing team, and an app support team that creates development tools and test benches, and supports customers that have issues. Customer product leads (CPLs) manage relationships with clients and bring enhancements requests and defect reports back from customers to the engineering teams. The CPLs are similar to the product owner role in scrum project management.

Previously, the CPLs used a spreadsheet to track work. Development was continuously prioritized by moving rows up or down within the spreadsheet. The engineering team used a separate system, developed in-house, to track defects found by tests and by customers. The firmware team had two backlogs of work: the spreadsheet and the issue-tracking defects. The engineering manager collected metrics by writing Visual Basic (VB) macros in Microsoft Excel to collect data from outside of the issue tracking system. They handled configuration management (CM) with an existing, branch-based tool.

Figure 1. Project's organizational structure
Elements and roles, issue tracking in the center

Business expansion

The economic downturn of the late 2000s represented an opportunity for the business to expand its customer base. Each customer had unique requirements and possible additional hardware needs. What began as a single product on one hardware platform grew to 18+ product variants across four hardware platforms. Figure 2 depicts part of that growth from a single Product A to variants across multiple hardware platforms. Each stream represents a product variant, and the arrows represent how changes might flow between variants.

Figure 2. Product's software and hardware variant flow
Product A flow diagram

The cellular communications industry is highly dynamic and agile. Manufactures are continuously integrating new versions of their suppliers' parts for feedback, to reveal issues early and show progress. Therefore, suppliers are required to frequently deliver new versions, sometimes multiple times a month, in response to that feedback. The CPLs are in constant communication with manufactures to decide what changes to include in the upcoming delivery. Based on those conversations, the CPLs re-plan work with the firmware team lead twice a week (twice-weekly sprints from an agile perspective).

Expanding customers led to large challenges for the firmware team for several reasons. Each new customer typically has custom enhancements. Not all changes (enhancements and defect fixes) are delivered to every customer variant because of factors such as cost of the feature, hardware platform, available memory, and so forth. To make matter worse, customers might choose to take changes in a different order. In the example illustrated in Figure 3, Customer A requires a change be delivered without the two preceding changes. The preceding changes might or might not be delivered to Customer A's variant sometime in the future.

Figure 3. Changes delivered out of order to Customer A
Third product line change delivered as second

Challenge to engineering and agility

The expanding customer base also challenged engineering to meet the selected variation and schedules of new customers. Managing change in this environment is complex, because not all changes will be delivered to all product variants, or they might be delivered in a different order. To track the work, each customer variant has its own backlog of prioritized work, as shown in Figure 4. Note that there is a single development team, the firmware team, that implements all of these backlogs.

Figure 4. Tracking work across product variants
firmware versions, flow to stories and defects

As new customers were added, their work backlog was recorded in a new tab in the Excel spreadsheet. Rows representing work (defects or enhancements in the form of user stories) were copied between tabs to record the fact that a particular customer was interested in a particular change. However, in a spreadsheet, there is no way to relate duplicate work as Figure 5 shows. There was no indication, for a given change, which product variant had the change and when it would be delivered.

The expanding customer base also challenged their legacy CM system. In branch-based CM systems, each file modified has to be placed on a separate branch, as shown in Figure 5. A change might touch 5, 10, or more than 100 files, and that requires the firmware engineer to create the proper branch for every file.

Figure 5. Existing change management approach to integrating product variants
Diagram of CM integration of product variantsDiagram of CM integration of product variants

Most merges in this environment are complex, because not all changes will be delivered to a particular customer or (worse) may be delivered out of order. Complex merges in branch-based CM systems are error-prone, because the CM systems do not manage a change; they manage only individual file variants. The engineer performing the merge is responsible for understanding which files are associated with the change, which branch those files are on, and where they need to merge. That's a time-consuming, error-prone process.

Merge errors are costly because they are often difficult to locate. Because of this complexity and potential cost, the firmware team had established a role called the Integrator. Only Integrators were allowed to merge changes to customer branches. Unfortunately, Integrators became a bottleneck in the process and sometimes caused delays in getting features or defect fixes into the releases that needed to be delivered multiple times per month.

Although not explicitly calling themselves "agile," the companies in the cellular communications industry follow many agile principles, as this one does. They frequently integrate changes (continuous integration), even across suppliers, multiple times per month. And based on what they learn from frequent conversations with the customer, they continually reprioritize their product backlogs (adaptive planning). For this organization, that happened twice weekly.

Agile practices of smaller work, adaptive planning, and continuous integration can overwhelm a development organization not prepared to handle these changes. Their existing tools were not designed to handle the frequent change and integration that we see today. In many cases, we now see that the organization's change process takes more time than the change within the system. The overhead of this change process is often the fault of the change management tools, as it was in this case: outdated CM system, Excel, VB macros.

In summary, this engineering organization faced three key challenges:

  • Excel was insufficient for planning and tracking work across product lines. The process quickly fell apart when they tried to manage and prioritize changes across multiple product lines and then tried to understand which changes had been made to which variants and when a particular change would be made to a variant.
  • The branch-based CM software was inadequate to meet their agile, continuous integration needs. There was no simple way to relate code changes to the work tasks in the spreadsheet plans, so it was difficult to see what changes others had made in previous product variants. The CM tool did not help with complex merges, which led to the need for an Integrator role to perform the merges. Integrators became a bottleneck and delayed delivery of changes.
  • The engineering manager wasted time creating VB macros in an Excel spreadsheet to connect to the development repository for status information

Agile adoption with Rational Team Concert

The team chose to adopt Rational Team Concert to address those challenges. The organization had three primary needs:

  • A more powerful configuration management tool
    They needed a solution that helped more easily identify all of the file variants to be merged when they were selecting a change into a product variant. It had to provide an easier way to integrate changes so there was no more need for the Integrator role. The software also had to support relationships between the Excel spreadsheet tasks and the code changes being made, so that they could better understand what code changes were made when reapplying that change in a different product variant.
  • Simplified planning across product lines
    They needed to create separate but related work tasks for a common change that would apply across multiple product variants, so they could be independently managed and prioritized. The work tasks needed to be related so that CPLs could tell which product lines had that defect or enhancement, and so development could locate all of the code changes for that defect or enhancement. And they needed a mechanism that would simplify their work scheduling within and across product lines.
  • Easier and better real-time visibility of development status
    The engineering manager was tired of updating and running VB macros to get the status of the development process.

A more powerful configuration management solution

This section describes how the team used Rational Team Concert to address their challenges relating to configuration management.

Relating code changes to work

To address the first need, Rational Team Concert provides automatic relationships with work items. Software developers can easily associate a collection of code changes with the work task assigned to them. The screen segment Figure 6 shows the work items and their associated file changes. This enables developers to quickly find out what changes were made on another product line without looking at the code branches. Before Rational Team Concert, engineers had to look at the code branches to find what someone else did on another product line. Now they simple locate the work task and see all associated changes sets.

In addition, developers can use Rational Team Concert to work on multiple changes simultaneously in the same workspace. They do not need to create multiple local copies of the source code repository and remember which local copy is associated with which change. The developer simply selects the work item to suspend or resume work, and Rational Team Concert automatically configures the local sandbox with those changes.

Figure 6. Working multiple changes in Rational Team Concert
screen shows Outgoing and Suspended work items

Stream-based configuration management

Rational Team Concert stream-based configuration management provides a simpler, more powerful approach to managing product line changes. Figure 5 showed the challenges of selecting an individual change and then finding and merging all related files. Rational Team Concert automatically identifies all file changes that must be merged into the incoming stream (product variant).

Figure 7. Integrating changes into products lines using Rational Team Concert Streams
Diagram, one change delivered to Customer A

One limitation is that Rational Team Concert uses a patch solution when change sets are delivered out of order. The challenge with the patch mechanism is that we lose traceability to a common change (however, you will see a solution to this limitation later in this article). The Rational Team Concert team understands that they need a better solution, and there are several work items in the Rational Team Concert backlog to address this issue. For more information, see the Rational Team Concert work item number 128329.

Help with complex merges

Rational Team Concert helps address complex merges in multiple ways. First, it provides a view of all the factors necessary to merge a change into a new stream (product variant). Therefore, there is no mistaking which files need to be merged. As Figure 8 shows, Rational Team Concert lists the files for each change set and indicates where there is a potential conflict in the merge.

Figure 8. Rational Team Concert merge
Snapshot of Rational Team Concert pending changes for complex merge

Developers can also take snapshots of their personal workspaces before a complex merge. During the merge process, Rational Team Concert automatically stores all merge-related changes in a separate change set. If the merge is not going well, the developer can revert back to the snapshot state by either suspending or discarding the change set.

Figure 9. Workspace snapshot
Screen capture

Multiple, linked tasks for work across product lines

Developers can duplicate a work item to create an independent copy with a reference to the original. Duplicate work items are useful for the firmware team to create the work performed across multiple product lines. The independent (but linked) work item can be separately planned and developed. You can quickly view the links on the original work item to see what other variants will have this change and whether they have been delivered yet.

In Figure 10, Story 174 is created as a duplicate of Story 57. When the developer begins work on Story 174, he can follow the link to the related Story 57 (highlighted in Story 174) and then accept Story 57's change set (highlighted in Story 57 and shown as the red change set) into his workspace. Accepting the previous story's change set automatically places all of the file difference into the developer's workspace. New changes required for the merge are automatically placed in a new change set and associated with Story 174, as shown in the figure.

Figure 10. Linking work across product variants in Rational Team Concert
Relationship between work items targeting different variants and their changes sets

Process support

The firmware team also uses Rational Team Concert process controls. Broken builds dramatically slow progress for agile teams that use continuous integration practices. Rational Team Concert enables teams to lock down delivery to streams so that only work that is currently planned for release to a customer can be delivered. Although someone might think that any working software could be delivered during an iteration, embedded software is sensitive to time and space issues, and software not planned for a current iteration can distract from the upcoming release and break builds.

However, the firmware team also wanted to enable engineers to collaborate and explore implementation solutions for future work. The solution was to enforce a rigid delivery policy for the customer streams and then create a Rational Team Concert sandbox team with more relaxed delivery rules. Developers who were members of the sandbox team could create additional streams for collaboration, using the less restrictive rules.

Simplified planning across product lines

Spreadsheet planning has many limitations. It is difficult to manage and track the duplicate work across product lines, reprioritizing is hard, and there is a general reluctance to create detailed work tasks because creating and moving them is so difficult. As a result, many work tasks associated with a change (updates to the hardware test bench, modification to development tools) are not tracked and can be lost.

Child work items

Rational Team Concert was one of the first issue-tracking systems to simplify relationships between work items. The software makes it simple to create relationships and then later navigate and query on them. Embedded systems development might require downstream tasks after firmware changes are made — updates to the test bench, customer development tools, user docs. You don't want to lose those tasks, of course.

Those tasks would not appear in the spreadsheet planning, so they were at risk of being lost. Rational Team Concert relations between work items allows engineers to define this additional work and track it to ensure that it gets completed before delivery.

Planning across product lines

Planning across multiple product lines had long outgrown the spreadsheet. The CPLs and firmware team needed a mechanism to show what work was planned for which iteration and to easily allow work to move between iterations.

Rational Team Concert agile planning capabilities solved many of these issues. Work can be viewed in many customized plan modes, enabling the CPLs (product owners) to view work for a single product line. The week after the new software was installed, the CPLs and firmware team leads used Rational Team Concert planning tools, rather than the spreadsheet, to drive the twice-weekly planning meetings, with great success.

Figure 11. Contrast spreadsheet and Rational Team Concert planning
Illustrations compare the two formats

Although it was successful, there are some challenges in using Rational Team Concert for planning across multiple timelines. The planning is based on agile principles. Timelines are decomposed into nested iterations (release-sprint) and, at any level, only one iteration is current (Release 2.1, Sprint 3). Agile teams work against a single timeline. To support these agile practices, the plans select a single team (or team of teams) and single iteration (release, sprint) for planning and tracking work. In product line systems, a single team is working multiple product lines, each with their own schedules.

There is a problem here where a single team (firmware) serves multiple backlogs, one for each customer variant. To see work for different customers, the firmware team needs to change the plan's Planned For property constantly. An agile advocate would argue that the CPLs should internally prioritize their work across customers and present a single, unified backlog to the firmware team. However, that is not a reasonable request, and there are Rational Team Concert enhancement requests to select multiple timelines on a single plan (for more information, see the Rational Team Concert work item 94056 ).

This organization chose to have the CPLs track the work in separate timelines. When the agile teams want to see their work, they change which timeline their plan references. Although not an ideal solution, changing the plan timeline was a good workaround, and the overall solution is significantly better than the spreadsheet.

Easier and better real-time visibility of development status

To report on engineering status and execution, the engineering manager wrote Visual Basic scripts in Excel to pull data from their previous issue-tracking system. Rational Team Concert work item queries and included dashboards provided a zero-effort solution for tracking progress. The engineering manager started configuring his dashboard the first week that Rational Team Concert was deployed.

Figure 12. Project visibility in Rational Team Concert
Screen captures of graphs and reports

Summary

This article shows how a telecommunications supplier developing embedded products across multiple product lines is applying agile practices. As business grew their customer base, the firmware team was challenged to meet delivery schedules due to a combination of factors:

  • Excel was insufficient for planning and tracking work across product lines. The process quickly fell apart when they tried to manage and prioritize changes across multiple product lines and then to understand which changes had been made to which variants and when a particular change would be made to a variant.
  • The branch-based CM software was inadequate to meet the agile, continuous integration needs. There was no simple way to relate code changes to the work tasks in the spreadsheet plans, so it was difficult to see what changes others had made in previous product variants for a particular change. The CM tool did not help with the complex merges, which led to the need for an Integrator role to perform the merges. Integrators became a bottleneck and delayed delivery of changes.
  • The engineering manager wasted time creating VB macros in an Excel spreadsheet to connect to the development repository for status updates.

Rational Team Concert resolved almost all of these issues with the changes shown in the table below.

Table 1. Changes resolved by Rational Team Concert
Change categoryBeforeAfter
Planning Microsoft Excel spreadsheets Rational Team Concert agile plans
Tracking duplicate work across product lines Try to locate across multiple tabs in the spreadsheet Link the related work items
Managing code for product variants Branches in the SCM tool Rational Team Concert stream per variant
Locating work performed in previous product version Find appropriate branch in the SCM tool View change set associated with Rational Team Concert work items
Who integrates code Special "Integrator" Everyone
Performing complex merges Error-prone, branch-based merge process Deliver change sets to customer stream, and Rational Team Concert identifies the required files to merge
Additional work for test bench, customer development, tools Firmware team "remembers" Rational Team Concert child work items
Status VB macros in Excel Rational Team Concert dashboards and plans

It is interesting to note that when first talking with this customer, branch-based integration was the issue they discussed. But they quickly realized that the Rational Team Concert work item and planning capabilities would be a much better solution for the product-line planning and tracking.

Resources

Learn

Get products and technologies

  • Download Rational Team Concert from Jazz.net and try it free on up to 10 developers for as long as you want (requires registration). If you'd prefer, you can try it in the sandbox instead, without installing it on your own system.
  • Evaluate IBM software in the way that suits you best: Download it for a trial, try it online, use it in a cloud environment, or spend a few hours in the SOA Sandbox learning how to implement service-oriented architecture efficiently.

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Rational software on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Rational, DevOps
ArticleID=900801
ArticleTitle=Being agile in embedded product-line development with Rational Team Concert
publish-date=04302013