An ancient Chinese proverb states that "The journey of a thousand miles starts with a single step."1 When embarking on a software metrics program, we don't want that journey to begin by stumbling on the first step. We want a metrics program that gains traction and succeeds, and avoids the pitfalls of another famous proverb: "If you don't know where you are going, any road will get you there."2
The Goal-Question-Metric Approach (GQM)3 is a powerful tool for taking the first step with confidence and direction. GQM helps us build the framework for a measurement model by putting the initial focus on goals, the objectives that we want to achieve with metrics.
In this article, I'll examine GQM from two perspectives. First, what does it take to apply GQM? I'll discuss the four phases for applying GQM: (1) establish goals; (2) define questions; (3) identify metrics; and (4) set the model in motion.
Second, what can we learn from knowledge workers of differing roles who have already applied GQM? I'll examine real-world GQM scenarios for IT executives, project managers, architects, development managers, and software quality managers.
I'll conclude by revisiting the nature of discovery and inquiry, and how GQM can do more than help us define first steps and beyond; how it can in fact help knowledge workers of very differing roles and perspectives to find common ground, enable and empower each other, and catalyze development governance and IT results.
Most people know GQM as the brainchild of Dr. Victor Basili, who wanted to create a conceptual framework for problem solving and brainstorming. The GQM approach to metrics is not limited to software development, or IT governance, or even technical problem solving, although it lends itself very well to technical environments. Essentially, GQM offers a framework for helping you understand whether or not your goals are being accomplished. You ask the questions that your goal demands, and you establish the measure that will verify the question's answer. GQM is flexible, designed to harness and direct our cognitive potential.
The GQMs presented in this paper are based on the interviews SourceIQ has conducted over the years with the people who are stakeholders in the outcome of their processes. These sets of GQMs are representative and typical, but they are far from exhaustive. Once you get the general idea, you'll be able to create your own sets of GQMs for your scenarios.
Phase 1: Establish Goals
Defining and articulating goals is a management imperative, so initiating GQM with goals is a natural and straightforward task. It is also a task that confers tangible benefit to the development organization. Goals clarify direction and reveal the road the organization will travel, whether around governance, quality, meeting schedules and budgets, or customer satisfaction.
Phase 2: Define Questions that Assess Progress to Goals
Once we've defined goals at a conceptual level, we then examine the operational questions that must be answerable in order to determine progress towards or achievement of goals.
The goals and attendant questions that we come up with are typically a function the roles we perform in the development organization. That is, they are based on the areas of interest and accomplishment for each type of knowledge worker.
When people with differing roles participate in GQM, their goals and questions may at first seem largely unrelated. For example, a CIO may want to drive innovation, contain costs, and improve communication with business unit counterparts. A project manager may want to improve estimation for future new and maintenance development activities. A QA manager may articulate specific goals and supporting questions around software quality and process stability.
However, as we progress into the third phase of GQM, we may be surprised to learn the degree to which seemingly disjoint goals for development governance are grounded in a common measurement model.
Phase 3: Identify Metrics that Support Answering Questions
Once we understand the questions needed to assess progress to goals, we then firm up the measurement model with the specific metrics required to form the answer to those questions.
Through empirical observation and hands-on work with customers, we've found that perceived differences in perspective and orientation that may come up in establishing goals and questions for different types of knowledge workers often dissolve when the measurement model is defined.
A typical outcome is that GQM develops a common understanding, with a common measurement model, and further that this model ties directly to the very human nature of software development: how much churn is there in our process? What is the volume and complexity of our projects? How many knowledge workers does it take to stay fresh and competent on intricate systems composed of many parts and pieces? Is the quality of the software work product compliant with engineering standards, and compatible with stable, robust systems in production?
Phase 4: Set the Measurement Model in Motion
Once we've applied GQM and arrived at a measurement model that supports our needs, we are ready to set that model into motion.
In "Development governance for software management"4, we discussed the role of an IBM® Rational® infrastructure in automating the collection and analysis of software metrics for compliance reporting. GQM helps us zero in on the metrics we need to achieve goals; Rational gives us the platform for delivering those metrics.
In other words, use GQM to determine what you will measure, and the meaning those measures have in assessing progress towards goals. Tap into the treasure trove of facts about your development process held in your Rational infrastructure, and start measuring!
Applying GQM by Role
GQM is often perceived as a top-down approach, which suggests the executive role is the place to start. But the driver, or potential driver, for change usually centers on projects. That means that project-level goals are an excellent place for GQM analysis to start. The analysis can expand upwards to how, say, a project meets longer-term corporate goals, or it can expand downward to how a project affects work for individual team members. I'll explore this a bit more at the conclusion of this article. Practically speaking, top-down should ideally meet bottom-up, which translates to organizational objectives and quality mandates coming down from the executive and management ranks, melding with a consensus on appropriate engineering standards and associated metrics coming up from the development and quality assurance ranks.
I strongly believe that GQM should not be perceived as a top-down sort of framework, which is sometimes described as a limitation of this approach. In fact, not only can the GQM concept originate in the lower echelons of an organization, it can also originate with a single role in an organization -- including an organization that may be resistant to the GQM approach to goal analysis. That is, it's not imperative that all roles in an organization get involved for the method to have benefit, although that's the ideal case. Any individual or team adopting GQM, regardless of their role, will benefit even if others don't buy in. (You will be better able to interface with your organization!)
Note: Some teams need GQM more than others. For organizations that are wholly new to this approach, start with the most needy teams, and don't try boiling the ocean in a large organization where change tends to come slowly.
The result of top-down meeting with bottom-up can be a harmonious meeting of the minds, or it can be a train wreck; usually it winds up somewhere in the middle. Let's explore some GQM relative to various roles in a development organization.
The IT executive will typically be a CIO reporting to a business unit. Goals at this level center on ways to retain and attract customers, be competitive, and reduce costs without compromising essential capabilities. Here are four examples of applying GQM to achieve these goals, which in SourceIQ's experience are most typical.
- G: Enhance delivery of innovation to customers.
- Q: What do we need to do to shift our focus and budget allocation from software maintenance to new development?
- M: Volume of code in production.
Rapidly growing or bloated code leads to spiraling maintenance costs. It just takes that many more people to stay current on the implementation, in order to fix bugs and add new features. Contain maintenance costs and reverse the trend by tracking the increase in the volume of logic produced. Ties to Architect goals below.
- M: Complexity of code in production.
Over time, an implementation will accrete complexity as code is revisited for bug fixes and feature enhancements. An executive focus on complexity influences team behavior by making streamlined, maintainable code a priority. Combined with volume, the result is lower TCO for systems, freeing up budget and personnel for new innovation.
- G: Improve relationship with business units by improved delivery, increased transparency.
- Q: How can we improve delivery, transparency and communication on progress toward goals with business unit counterparts?
- M: Volume of code in production.
By communicating the volume of logic assets for business critical applications, and the rate of growth of those assets in conjunction with recent maintenance and enhancement activities, IT is able to engage with customers in a realistic, fact-based manner. Business unit counterparts gain an appreciation for the scale of the undertakings, and the significant efforts performed on their behalf.
- M: Volatility of software development activities.
Visualizations of the churn taking place with business logic show IT's responsiveness to business unit requirements, enhancement requests and maintenance activities, yielding transparency to development activities that are otherwise opaque.
- G: Obtain baselines for more accurate budget forecasting.
- Q: How can the actual effort required to accomplish past tasks and milestones be assessed and applied to future undertakings?
- M: Volume and volatility of software development activities.
The volume and volatility measures associated with past efforts form baselines for future implementations of similar scale and scope. Assuming that the skill sets and domain expertise for development personnel remain relatively constant, and that the project's applied technologies have not altered dramatically, the transactional information contained in your software configuration management (SCM) systems can be mined to derive clear and consistent baselines for future efforts. What's more, management should also be able to see measurable improvements against baselines in relation to progressive application lifecycle management (ALM) initiatives (training, test automation, continuous integration, Agile methods, reduced requirements churn, use of dynamic languages, etc.)
- G: Obtain superior price/performance from development partners.
- Q: How do my partners compare relative to my organization's standards, and to one another?
- M: Volatility, complexity, language rule compliance, coding guidelines. Managing external development can pose significant challenges. Is there adequate service level agreement (SLA) compliance? Is the activity taking place compatible with company standards (complexity, and other compliances) and scheduled release dates? Is effective knowledge transfer in place? Is one partner superior to others for certain types of projects or tasks? Use metrics to form apples-to-apples comparisons, and drive partner relationships to superior delivery and efficiency.
A typical project manager (PM) faces the chronic problem of being removed from the day-to-day details of his or her project. At the same time, project managers worry about incorporating quality goals into production, and meeting any number of scheduled dates. These concerns magnify as new modes of production get introduced, such as agile development methods combined with globally distributed teams -- these realities quickly lead to thorny scenarios. What PMs need are consistent usage models, progressive modes of team empowerment, and measurement models that support the adoption of new goals.
- G: Ensure stable, predictable development processes that meet scheduled milestones.
- Q: Is my project's trajectory compatible with accomplishing tasks for scheduled milestones?
- M: Volatility of project-related software development efforts (branch, stream, Unified Change Management (UCM) activities).
A project's trajectory is measured by the effort and efficiency in transitioning through key stage gates. In the IBM Rational Unified Process® (RUP®), a PM is keenly interested in what happens from elaboration through construction; in more traditional terminology, from a design phase to a coding phase. When a project is on track, the volatility of its development efforts correspond in scale and duration to the project plan, and that volatility follows an observable glide path arriving at a stable landing for each scheduled release point. Deviations from the project plan are readily detectable as variances in the volatility, and tell the PM to apply a course correction to enable the team to get back on track.
- G: Maintain the appropriate personnel allocation to meet scheduled milestones
- Q: Is my global team staffed properly, and are the people contributing effectively to meeting the project objectives?
- M: Volatility of project-related software development efforts (by individual or working group, per branch, stream or UCM activities).
In order to meet project schedules with confidence and predictability, PMs must maintain open and effective communication between team members, and understand when the personnel mix needs to be adjusted. This can be tricky even in a simple bullpen setting, and far more difficult as the team is staffed globally. PMs can assess the contributions to SCM systems made by individuals and working groups, and determine whether people are able to complete their task assignments in a timely manner. This assessment is ideally made by coupling task assignments from the project plan with the development methodology (lightweight or heavyweight).
- G: At project completion, deliver code into production that meets organizational quality goals.
- Q: Is the work performed at every stage of development compatible with meeting this goal?
- M: Complexity, language rule compliance, coding guidelines.
By monitoring these metrics, a PM can establish a more effective working relationship with the project's Development Manager and Software Quality Manager. This relationship is fundamental to identifying quality issues upstream in the development process, when they are easier and less costly to fix, so that the overall quality trend lines remain consistent with production quality standards.
Iterative development methods typically place architects at the beginning of a project, partnering with development teams through requirements and early bits of coding. Keep in mind that this is true not only for your project, but often for a dozen other projects as well. The need for architectural uniformity across projects means that when your team starts cranking out the code, the architect is frequently hard at work on the Inception phases of some other project. What architects need is some level of visibility into all projects to ensure that the goals of senior leadership are being addressed across a busy corporation. If each software development team is going off in their own directions, chaos will ensue.
- G: Ensure that architectural constructs are being applied properly as projects move through elaboration and construction
- Q: Is the profile of the technology stack being implemented by the team compatible with the architectural goals (componentization, service-oriented architecture (SOA), frameworks, etc.)?
- M: Volume of logic artifacts by type, complexity.
Architects can lose visibility into a project once it moves through elaboration (design) and into construction (coding), often because they have moved onto other projects, or for some other reason are at arm's length from the coding team. It can happen that the team will stray from the architectural principles that were intended to lead to a more robust, maintainable system. Design rules are forgotten, interfaces eroded, and technology stacks become a mile wide and an inch deep. By monitoring the volume of the logic artifacts produced during development, architects gain a deep and accurate sense of the patterns being applied by developers, and are better equipped to nudge the team back on track when it starts to stray. Complexity can also yield deep insights into the effectiveness of a design and the translation of that design into code.
- G: Ensure that the organization is making an appropriate use of technologies.
- Q: What development technologies are currently in use, and is the actual outcome of that development consistent with what was planned?
- M: Volume and volatility of logic artifacts in production by type.
Companies adopt technologies that they believe are best suited to achieving successful systems in production with the available project personnel and budgets. By monitoring the metrics associated with logic for production systems, architects can determine whether these decisions are working out. For example, it may be that what looked like a great development approach yields far too much code, requiring too much senior development expertise, to be maintained by the available team. By participating in a fact-based discussion on applied technologies, architects are able to help guide the organization towards object technologies, dynamic languages, declarative languages, design patterns, frameworks, and other constructs that help teams, executives, and managers achieve their goals.
The development manager, or lead developer, is a tough job these days. They're talented, which means they work longer hours. The ones I've met all describe the difficulty of getting team work done, and at the same time managing the organizational churn caused by high turnover on domestic teams, and really high turnover on offshore teams. Inexperienced junior developers need to be paired as much as possible with senior developers to maximize productivity. As junior team members improve, development managers have more time to improve the code they're responsible for.
- G: Maximize productivity of all team contributors.
- Q: Are developers able to complete task assignments, or are they hitting roadblocks?
- M: Volatility of project-related artifacts by individual or working group.
One of a development manager's greatest challenges is helping developers to get past roadblocks. It's often human nature for people to clam up when they're having trouble doing their job, and certainly most developers don't want to look bad in the eyes of their manager. Like the rest of us, developers will also sometimes procrastinate on tougher assignments. Regardless of the root cause, these very human tendencies can impact programmer productivity, and jeopardize a project's success. Development managers can monitor the contributions of developers -- in the bullpen or on a different continent -- relative to task assignments and methodology. When developers become stalled on a task, the manager can intervene in a positive and constructive fashion, and help get things back on track.
- G: Establish focused mentorship for junior developers.
- Q: Which developers need help in what areas?
- M: Complexity, language rule compliance, coding guidelines.
Development managers are responsible for helping junior developers to mature into senior developers. By monitoring the work of junior developers against organizational standards, development managers are able to see who needs mentorship and in what areas. Development managers can then pair junior developers with senior staff who can give them the learning experiences that they need.
- G: Reduce implementation risk by prioritizing peer review around high-risk code.
- Q: What modules are most likely to contain errors, especially those that will be hard to find in test, or that will be the most difficult to understand and maintain?
- M: Complexity, language rule compliance, coding guidelines for latest code release (label).
Peer review is a powerful tool for sharing knowledge of an implementation, and streamlining it to be more robust and maintainable. As systems grow in scope and size, it becomes increasingly difficult to prioritize which modules to review. Metrics can pinpoint the modules that (a) are most likely to have incomplete test coverage, (b) are intrinsically error-prone, (c) perform poorly or fail outright according to the language vendor, and (d) consume a disproportionate amount of time and effort to maintain.
Software Quality Manager
Arguably, the first to benefit from a GQA approach are the quality assurance (QA) teams. Being able to partner with other roles in the lifecycle and increase efficiency are genuine QA goals, and GQM puts the power of a measurement model into QA hands. GQM can help quality become part of each phase of development, not just at the end. QA people understand this, but developers and architects frequently do not. GQM-based questions and answers can introduce new constructs that support quality goals, leading to fewer flaws and a smoother process overall.
- G: Maintain consistent baselines for QA reporting.
- Q: How big is a given release (input to calculating defect density)?
- M: Volume.
The more discerning QA can be in establishing the size of a release, the more useful that defect density becomes. Size is typically given as Lines of Code (LOCs); consider using eLOC as the LOC metric that best represents the true volume of logic. When using LOC, it is important to count only those artifacts that contribute logic to test. It can also be very helpful to distinguish between types of artifacts that call for different test strategies (e.g. Java™ versus JavaServer Pages (JSP)). Other measures of size include number of routines, statements, and function points.
- G: Increase QA efficiency and contain costs by avoiding unwarranted testing.
- Q: Is the current state of development compatible with initiating test?
- M: Volatility.
Testing software can be time-consuming and expensive, especially as software systems are increasing in scope and complexity. Volatility helps QA managers understand if a release is stabilized to the point where it can reasonably be subjected to test.
- G: Be a more effective partner with Development on maintaining development governance.
- Q: Do the releases submitted for test comply with established development standards?
- M: Complexity, language rule compliance, coding guidelines.
QA plays an integral role in achieving development governance. It is the final stage gate in auditing a release's compliance with development standards. The onus is clearly on Development to build the quality into software systems, since quality can't be tested into software after the fact. QA can implement a valuable and appropriate audit function by applying a "trust, but verify" model to development standards.
GQM can be an invaluable aid in helping you to take the first step in a software metrics program. Whether for governance, software quality, or process improvement, defining a metrics program based on goals ensures the relevance and value of your effort, and makes a journey of a thousand miles worth taking.
Let's close by clarifying a couple of key issues. Is GQM truly top-down? Is core metrics definition truly bottom-up? When companies try to adopt something new, there's usually a degree of organizational resistance. We don't like change. The rank and file get perturbed, even frightened, if they think some new top-down management scheme is being forced upon them. What Basili pioneered with GQM, however, is a way to empower individuals and their teammates wherever they may be in the organization -- whether at the top, at the bottom, or in the middle.
We all have goals pertaining to our role. Being respectful of the process and the goals of others makes us learn how much we have in common with other roles.
Fifteen years ago, Jeff Alger published a useful clarification, which he called "center, periphery, calibrate." The idea is that people begin by focusing on the familiar, then move out of their comfort zone to explore new ideas and concepts, and finally recalibrate their perspective. Iteration upon iteration leads to an expanded view of the world.
With GQM, we define goals based on our respective roles, or centers. Each of us expresses goals based on our center. For example, at the outset, we may view our goals in isolation, and not as part of a larger whole, as shown in Figure 1.
Figure 1: At first, team roles may feel isolated from other roles.
As the GQM exercise progresses, awareness grows of other organizational goals. With awareness comes understanding of peripheral issues, and then recalibration bring each of us into closer alignment with the organization, as shown in Figure 2. The union of those goals represents the true organizational objectives, so it is important to organize and conduct GQM in such a way that the full complement of goals can be incorporated into a consistent vision and purpose.
Figure 2: With understanding comes awareness of peripheral issues, which connects each role more closely to others.
Ultimately, the process is neither strictly top-down nor bottom-up, but rather a coming together of governance objectives supported by a well-defined set of standards that can be assessed for compliance.
Consider GQM in terms of the practices outlined in "Best practices for lean development governance" : iterations, adaptation, and continuous improvement. When a pragmatic governance body uses GQM to facilitate a program of simple and relevant metrics, the critical first step can be taken with confidence and direction.
- Proverb by Lao Tzu, Tao Te Ching
- Quotation by Lewis Carroll
- Multiple citations; see The Experience Factory (Basili and Rombach) at http://www.cs.umd.edu/~basili/publications/technical/T86.pdf
- "Development governance for software management: Using IBM Rational tools to assess process variability and standards compliance" (Dunn), The Rational Edge, October 2007.
- "Best practices for lean development" (Kroll and Ambler), The Rational Edge, June/July/August 2007.
- Participate in the discussion forum.
- A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
- Global Rational User Group Community