Automation for the people: Continuous feedback

Receive immediate feedback with every source code change

Feedback is vital for the practice of Continuous Integration (CI) -- in fact, it's the life blood of a CI system. Rapid feedback enables speedy responses to build events that require attention. Without feedback mediums like e-mail or RSS, builds in a broken state have the tendency to stay broken, which defeats the purpose of CI in the first place! In this installment of Automation for the people, automation expert Paul Duvall examines various feedback mechanisms that you can incorporate into CI systems.

Paul Duvall (paul.duvall@stelligent.com), CTO, Stelligent Incorporated

Paul DuvallPaul Duvall is the CTO of Stelligent Incorporated, which helps companies address software quality with effective developer testing strategies and Continuous Integration techniques that enable teams to monitor and improve code quality early and often. He is a contributing author to the UML™ 2 Toolkit and currently co-authoring Continuous Integration: Improving Software Quality and Reducing Risk (Addison-Wesley).



14 November 2006

Also available in

When I describe CI to technologists that haven't used the practice yet, I tend to focus on one of its major benefits: a decrease in the time between when a defect is introduced and when it is fixed. This benefit is a direct result of a CI server's ability to provide rapid feedback of build statuses. In my opinion, rapid feedback of build events is the central tenet of CI and one that causes many to describe a CI system as a continuous feedback mechanism.

When a build is broken, I want to know immediately so I can take the proper action to fix it -- whether that be a compilation error, test failure, or even an increase in overall complexity. The sooner I know, the more relevant the information and the better chance I have to fix the problem.

Feedback, however, is useless without action. In the case of CI, this action must be prompt because a broken build affects everyone; therefore, the feedback mechanism employed by a CI server must be timely. The beauty is that most, if not all, CI servers available today provide an e-mail feedback mechanism; however, there is a cornucopia of other mediums to choose from ranging from Really Simple Syndication (RSS) feeds to SMS text messages to X10 devices. And these various feedback mechanisms quickly facilitate jumpstarting the necessary people into action.

About this series

As developers, we work to automate processes for users; yet, many of us overlook opportunities to automate our own development processes. To that end, Automation for the people is a series of articles dedicated to exploring the practical uses of automating software development processes and teaching you when and how to apply automation successfully.

Execution a la e-mail

The ubiquitous nature of e-mail makes it a simple and useful means of feedback -- every CI server I'm familiar with provides some type of e-mail feedback mechanism. In fact, most CI servers provide the capability for configuring who receives which message, based on a desired build event. For instance, a technical lead may always receive an e-mail regardless of build status, but the project manager only receives an e-mail when the build fails. Additionally, the developer or developers who checked any code into a CM system (and thus triggered a CI build) always receive an e-mail indicating the build's status.

Listing 1 is an example of a CruiseControl main configuration file (config.xml), which is configured to send an e-mail, in HTML format, to the project's "build master" for both successful and failed builds:

Listing 1. Sending an e-mail with CruiseControl
<publishers>
  <htmlemail
    mailhost="localhost" 
    defaultsuffix="localhost"
    username="bfranklin"
    password="G0Fly@Kite"
    returnname="Buildmaster"
    returnaddress="buildmaster@localhost"
    subjectprefix="[CC]"
    xsldir="webapps/cruisecontrol/xsl"
    css="webapps/cruisecontrol/css/cruisecontrol.css">
    <always address="buildmaster@localhost"/>
  </htmlemail>
<publishers>

You can use the defaultsuffix attribute of CruiseControl's e-mail task to also send e-mails to the developer(s) who committed the most recent changes, as well.

Figure 1 shows an example of the e-mail a user will receive based on the configuration in Listing 1:

Figure 1. A sample build status e-mail a la CruiseControl
A sample build status e-mail a la CruiseControl

While e-mail is arguably the de facto standard feedback mechanism for projects that use CI, if team members are flooded with too many e-mails, they may begin to ignore them, which, of course, defeats the purpose of a feedback mechanism in the first place. I find it more productive to configure CI systems to send e-mails only to the developer or developers who applied the most recent change to a code base and to project leads who don't mind receiving a flood of build status e-mails every day.

What is Continuous Integration?

Continuous integration (CI) is a practice that enables teams to receive feedback and make improvements on a continual basis rather than waiting until later in the development cycle to find and fix defects. Tools (like CruiseControl) run in the background polling a version control repository looking for changes. When a change does occur, the tool executes a predefined build script through Ant, for example. Continuous Inspection is facilitated by the practice of continuous integration

RSS to the rescue

RSS is both a passive and active form of feedback, which can make this feedback mechanism more appealing because of a reduction in the amount of e-mail messages you'd receive otherwise. An RSS feed is a generated XML file (which conforms to a standard) that is updated based on certain events. An RSS reader then picks up the change and creates a passive message indicating new content. So, if you'd rather not get bombarded with e-mail, try this approach instead.

Within a CI system, this RSS feed can be updated based on the latest build status. For instance, Figure 2 is an example of the CruiseControl RSS XML that, by default, is generated for any build event:

Figure 2. RSS XML from CruiseControl
RSS XML from CruiseControl

Simply point your handy RSS reader to this feed and when a new build event occurs, and the RSS feed is updated, bingo, you'll be notified. Figure 3 shows the RSS feed as displayed in an RSS reader:

Figure 3. RSS indicates things went wrong!
RSS indicates things went wrong!

Stir it up with SMS

What if you're away from your computer and the build fails? If you believe that keeping an integration build successful is one of the highest priorities on your project, you'll want to know the build status at any point in time. One way to constantly keep abreast of build statuses is through the Short Message Service (SMS), which are short text messages sent to mobile phones.

Fortunately, the same mechanism used to send e-mail can also be used to send SMS text messages. For example, Listing 2 shows an example of using the CruiseControl e-mail task to send an SMS message:

Listing 2. SMS sample code
<publishers>
  <email mailhost="localhost" 
    returnaddress="buildmaster@localhost"
    returnname="Buildmaster"
    reportsuccess="fixes"
    subjectprefix="brewery"
    buildresultsurl="">
  <failure address="7035551212@vtext.com"/>
  </email>
</publishers>

Why so many feedback mechanisms?

When using CI, the criticality of a broken build is paramount and many teams do not move to other tasks until the integration build is successful again. Accordingly, it's often effective to choose multiple mechanisms that serve different purposes, such as SMS for those who aren't at their computers often and X10 devices that enable people to know a build's status in a glance. It is probably ineffective to use all the feedback mechanisms I cover here on a single project-- choose what works best for you and change it up every now and again to add some spice to a project, so to speak.

I have configured the e-mail task in Listing 2 in such a manner that I will receive text messages only when there is a build failure, which is an effective means to reducing SMS messages. Furthermore, the reportsuccess="fixes" attribute of the e-mail task indicates that you will receive a follow-up text message when the build has been fixed.

Figure 4 shows an example of the message sent from the CruiseControl server when a build fails:

Figure 4. Text message on your phone -- has your build failed?
SMS on mobile phone with build failing

Using SMS to send text messages can be an effective form of feedback for urgent build status messages. Generally, it is not recommended that you send text messages every time a build occurs (especially if you pay per message!). Remember, too much information can become "noise" that reduces the effectiveness of feedback, so choose wisely.


Excite things with X10

E-mails work well and SMS is great when you're away from the office, but what if you just want to glance at something to determine a build's status? Luckily, there is an entire cottage industry around programmable devices, dubbed X10 devices, which can provide a more passive form of feedback when hooked up to CI systems. With X10-enabled devices, you can control most any electrical device that can be enabled using special hardware that can be manipulated through the X10 protocol. This means that every day devices such as lights can be used to notify us of a build failure.

For instance, Listing 3 offers an example of the code required to plug an X10 device into CruiseControl's publisher mechanism. This example demonstrates turning on a red warning light if the build fails. The light is connected to an X10 controller (several "X10 kits" are commercially available). CruiseControl includes the Java™ COMM API so that you can use CruiseControl's config.xml to communicate with the device and turn it on or off. The X10 controller will connect to a communications port on your build machine; the port attribute indicates the communications port you are using. The houseCode and deviceCode attributes refer to the letter and number you set on the X10 controller for the device. The onWhenBroken attribute indicates that the device will turn on when the build fails (and off when successful). Finally, the interfaceModel attribute refers to X10 computer interface. The default is CM11A.

Listing 3. X10 configuration code for CruiseControl.
<publishers>
  <x10 port="COM1" houseCode="A"  
  deviceCode="3" onWhenBroken="true" interfaceModel="CM17A"/> 
</publishers>

Now, when this publisher is utilized, a device, such as the one shown in Figure 5, will turn on when the build is broken:

Figure 5. an X10 red warning light
X10 red warning light

There are a variety of X10 devices -- LavaLamps, sirens, you name it -- that can really enliven a project's environment. Adding these devices to a room is bound to create excitement, along with a clear indication of a build's status.


Don't forget about monitors and IM

If you'd rather receive feedback instantly, have a look at client monitors and instant messaging. A client monitor is an application that is always running in the background and periodically polls your build server for the latest build status. Of course, everyone knows what instant messaging is, and a host of popular platforms, from AIM to Yahoo to MSN, are available for this purpose.

Monitoring status with monitors

CruiseControl.NET (a Continuous Integration server for the .NET platform) provides a monitoring mechanism called CCTray that works with both CruiseControl.NET and the traditional Java built CruiseControl and is run from the Windows taskbar (which means it works only on Windows machines). An example of CCTray in action is shown in Figure 6:

Figure 6. CCTray monitor from the Windows taskbar
CCTray monitor from the Windows taskbar

You can use another client monitor called Nag on non-Windows machines and even on other CI servers, such as Apache Continuum.

Instant feedback with IM

You can use instant messages (or IMs) to notify project members of the status of the build too. Just like using a monitor, it's another way to receive feedback quickly because of IM's ubiquity among developers.

CI servers like Apache Continuum provide a simple way to send IMs regarding a build's status. For example, Figure 7 displays an example of a failed build message, from Continuum, which is sent to a Google Talk instant message client using Jabber. CruiseControl also provides a Jabber publisher to send messages.

Figure 7. An IM signifying a failed build
Figure Description

Monitors and IMs are some of the quickest ways to send build status feedback, and I'm familiar with project teams that exclusively use these mechanisms. As long as this is coupled with another way to inform project members when they are away from their desk (such as with e-mail or SMS), this can be a highly effective form of feedback.


A party of feedback mechanisms

As you can see thus far, there are many types of feedback mechanisms available for you to utilize in a CI environment. Many other feedback mechanisms, which I did not cover, are available, including the Ambient Orb, Web browser plug-ins, sounds, and widgets. Table 1 provides a summary of the feedback mechanisms I covered in this article, plus a few others I didn't:

Table 1. Continuous Feedback Mechanisms
Feedback MechanismDescription
E-mailProvides build status at discrete points in time
RSSPushes alerts regarding build status to an RSS reader
SMSAlerts you to build status with text messages sent to your cell phone
X10Provides feedback through visual means; popular examples include lava lamps and red flashing lights
Ambient OrbProvides another visual means of assessing build status; can be customized for alerts based on specific information
SoundsAlerts to build status through sound; especially useful for co-located teams
DisplaysProvides feedback through an LCD monitor
WidgetsDisplays the build status on the user's desktop; Yahoo! and MAC OS X provide Widgets for this purpose, including CruiseControl Dashboard for Java
MonitorAlerts you to the build status from the Windows taskbar
Instant MessageAllows for instantaneous notification
Browser plug-insNotifies you to build status through the browser; similar to CCTray, a Firefox plug-in for CruiseControl displays a greed or red icon for success or failure, respectively

The most effective way to proactively generate action is to use multiple feedback mechanisms in concert. Choose the combination that's most appropriate for you.


Have fun with it!

Remember, the purpose of build feedback is to initiate rapid action so that teams can fix a broken build. Typically, you'll need a combination of these feedback mechanisms to elicit this action. For instance, you may provide e-mails for all build events, SMS for build failures (and follow-up fixes), and an X10 device for visual notification. As your team matures in using these devices, you'll want to change it up every so often to add variety and keep people interested in build statuses. Be creative and have fun with it -- it's amazing how adding a few feedback gimmicks can enliven the mood and generate interest in the information they are providing!


Download

DescriptionNameSize
CruiseControl configuration for e-mail and SMSj-ap11146.zip1800KB

Resources

Learn

  • Automation for the people (Paul Duvall, developerWorks): Read the complete series.
  • "Bubble, Bubble, Build's In Trouble": Mike Clark, author of Pragmatic Project Automation, demonstrates a detailed example of configuring an X10 device for CruiseControl.
  • "Jabber" (Gerhard Poul, developerWorks, May 2002): Gerhard Poul shows how XML-based Jabber fits into today's e-business infrastructure, highlighting instant messaging in a whole new way.
  • "eXtreme Feedback for Software Development" (Alberto Savoia, DeveloperTesting, April 2004): Other examples of Continuous Feedback Mechanisms -- or as the author describes them: eXtreme Feeback Devices.
  • "Pragmatic Automation": Mike Clark, author of Pragmatic Project Automation, covers various feedback devices to monitor build status.
  • "Continuous Integration" (Martin Fowler, martinfowler.com, May 2006): A description of the purpose and practices of Continuous Integration.
  • "Configuring Continuum for CI" (Testearly.com): An article that describes the basic setup and configuration of Continuum.
  • Ambient Orb Ant task example (Testearly.com): Examples on using the Ambient Orb Ant task.
  • The Java technology zone: Hundreds of articles about every aspect of Java programming.

Get products and technologies

  • CruiseControl: An open source server for Continuous Integration.
  • 4-Pack Firecracker Automation System: Hardware to help configure your build to interface with X10 devices.
  • Nag: A Continuous Integration montior that works with multiple CI servers and platforms.
  • Yahoo! Widgets: Search for CruiseControl widgets for feedback to your desktop.
  • Ambient Orb Ant task: A freely available open source Ant task for changing Ambient Orb colors based on build status.

Discuss

  • Improve Your Code Quality discussion forum: Regular developerWorks contributor Andrew Glover brings his considerable expertise as a consultant focused on improving code quality to this moderated discussion forum.

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into Java technology on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=Java technology
ArticleID=174510
ArticleTitle=Automation for the people: Continuous feedback
publish-date=11142006