| New forum for Rational Edge readers |
|---|
| At the end of this article, you'll find a link to a new forum created specially for readers of The Rational Edge ezine. Get ready to add your thoughts about this article or other topics you've found in our pages. |
For organizations looking to build in quality and efficiency throughout the software development lifecycle, the realm of build management is among the last frontiers. The process of assembling all the pieces of a software system into a deployable, consumable product entails many steps and involves multiple disciplines, including development, configuration management, testing, and release management. While the activities within those disciplines might be largely automated, the interactions between them often are not.
On smaller projects this lack of cross-discipline automation might not be a debilitating problem. But as applications grow more complex, multi-platform, and/or service-oriented, and as teams become larger and geographically distributed, today's homespun and largely manual build processes start to unravel. The tasks you must perform to build and test your product become increasingly unpredictable and difficult to repeat. Risks to costs and schedule can mount to unacceptable levels.
It is generally recognized that development teams often spend significant amounts of time diagnosing errors that crop up during the build process -- sometimes more than they spend fixing the code once the source of an error is isolated. Likewise, teams often expend inordinate amounts of time maintaining and fussing over their script-driven build systems. This level of manual intervention inevitably hampers team productivity and reduces product quality. The question is: what can be done about it?
In this article I'll look at how the build management process often works today, and explore alternatives for automating it effectively across enterprise-scale development efforts. Lastly, I'll discuss the key capabilities of IBM® Rational® Build Forge™ for automating build and release management in environments of any size.
Caught in a cross-fire of escalating industry trends, development organizations are being asked to meet goals that seem mutually exclusive: increase product quality and accelerate time to market, even as software products and software development environments both get rapidly more complex.
There are many drivers compounding these challenges, the key ones being:
- The emergence of service-oriented architectures (SOAs) and web services, which integrate multiple, reusable software components to create larger and more sophisticated applications
- Intensification of regulatory compliance mandates, which may necessitate detailed audit records of activity across the development lifecycle
- A vast increase in distributed development, including various forms of outsourcing, as a means to both reduce costs and compress time-to-market
Thanks to these and other factors, development teams must become more efficient, more nimble, and more quality-oriented. This inevitably means enhancing the efficiency and effectiveness of process automation and integration in the software development lifecycle. The area more and more organizations are targeting for automation is build management.
It's not hard to see the general need for improvement in the build process. Being sandwiched between the development and release management disciplines, build management activities can fall through the cracks between existing toolsets. Each discipline involved in putting together a build (development, configuration management, testing, release management, etc.) uses its preferred tools for its own activities and often has a mature, well-structured process in place. But what is lacking is the quality of integration across all those disciplines. This is problematic because proficient build management is crucial to overall team efficiency. If you can't "turn the build crank" frequently enough, both development and testing suffer.
Methodologies like the IBM Rational Unified Process® (RUP®) offer best practices guidance for coordinating development activities across disciplines, but up until now little has existed in the way of solutions to help make that guidance operational on an enterprise scale.
Typical build management shortcomings
Many build management systems are essentially homegrown and have evolved over time. Teams often automate their build and release processes through scripting; however, these script-driven systems rarely scale across the increasingly large, complex, and distributed environments that characterize modern software development. Often, too, each task operates independently so wait times need to be built in and properly coordinated -- usually manually. In this kind of scenario, a broken nightly build can easily mean that QA staff will sit idle the next day, losing precious testing time while the development and build teams troubleshoot the errors.
Often there's a large margin for error when one function hands off a deliverable to another. Team members within each discipline generally operate somewhat autonomously relative to the other disciplines, in the sense that their focus is on their core competency, not on the linkage between their work and the overall process of building the product. As a result, sometimes the left hand might not know what the right hand is doing when the time comes to build the product. The lack of a consistent, repeatable process can lead to difficulties reproducing builds. This is particularly true in cases where processes are not adequately documented, so that knowledge about them resides with just one or a few people. Sorting everything out in the context of broken builds can thus be very inefficient. Build results are difficult to interpret and much time is spent chasing down errors.
Likewise, script-driven build management systems cannot make optimal use of build server resources, because often activities are hard-coded to run on specific systems. In the same way, hard-coded scripts make it more difficult for processes to shift so work can be shared or reallocated across distributed teams. Project risk increases inordinately if a key team member who knows "how to work the build system" leaves the project.
The limitations of open source
The reason so many development organizations rely on homemade, script-driven build management processes is not that tools for automating build management don't exist -- it's that they don't offer adequate scalability. Many open source build management tools, for example, work very well for small projects, which is what they are designed for. For example, they are generally geared for a single-server versus distributed, multi-server environment.
Organizations that have tried to scale their use of these tools up to large, distributed teams engaged in multiple projects have found that they invariably fall short in areas like process control, connectivity and auto-documentation support. The effort required to customize and maintain these systems for enterprise development environments often proves to be substantial. (I'm personally aware of one enterprise development team that put five developers on the task of rebuilding some open sources tools in an attempt to make them more scalable.) These activities add to the manual effort teams must perform, and steal valuable developer time from critical business priorities. In the end, many organizations find the limitations of most off-the-shelf build management tools put them, at best, on a par with do-it-yourself systems.
This lack of adequate tools has proved to be a stumbling block towards the adoption of more efficient practices by large teams. To evolve into an efficient product delivery organization you need supporting technology that scales across your enterprise. Similarly, you need auditable, repeatable, reliable process automation that you can codify and reuse across project after project. The drain on resources required to make (and re-make) your own tools for each job is generally too great. It would be far more cost- and time-efficient if such organizations could find acceptable third-party tools instead of building and maintaining custom software.
Benefits of a structured approach
A perennial challenge in creating higher quality software faster is the need to balance speed and efficiency (and also, particularly in the case of development, creativity) with adherence to a consistent, reliable -- i.e., structured -- approach.
Whether their mission is development, testing, or code control, teams do not want constraints on how they operate, particularly in their choice of tools. Efficiency within their discipline is paramount. Yet, seen from the wider perspective of the organization as a whole, optimizing efficiency might well entail some additional structure.
Take the issue of regulatory compliance, for example. Many businesses in industries like financial services and health care are under significant pressure to document how they do things like develop software, both within and across lifecycle activities. Documentation takes time and requires communication, potentially reducing efficiency. But the risk of sanctions can be significant to the company overall. Could your team produce documentation describing exactly how it built a product that was released a year ago? Could it reproduce that product during a compliance audit? Likewise, could it reproduce the build/release environment (the operating systems, libraries, server memory configurations, etc.)?
Ideally, you'd like to have just enough structure in place to be able to handle issues like these straightforwardly, but still have the flexibility to use the latest and greatest, best-of-breed tools within each of your individual development disciplines. That would be one way to empower development teams to deliver software faster while also making the development process -- and the build process specifically -- more reliable, consistent, and repeatable.
One way to appreciate what "just enough structure" means in a build management system is to step back and identify the cross-discipline capabilities that are required to optimize the quality and efficiency of build and release activities, irrespective of how each discipline accomplishes its particular mission.
Think of a scalable, efficient build and release management infrastructure as analogous to a modern, interstate highway, and the software development disciplines as cars on the highway. Whatever kind of car you're driving, be it a sports car or a minivan, the highway functions efficiently, in the sense that it gets you where you need to go. If you have a faster car you might get there faster, but the infrastructure supports you either way, and it does not dictate what kind of car you choose to drive.
Similarly, no matter what tools you use, you want the outcomes across your process to be consistent. Let's take a look at how a robust, structured build process can help provide that consistency.
Perhaps the key capability that you get from a scalable, cross-disciplinary build management process is better visibility across development activities. This is extremely valuable because when something breaks, you can see more quickly what happened. If a build keeps breaking at a particular point, for instance, you can readily identify what discipline to look to for the cause, because your process is consistent and well-defined. Then you can drill down into the area where the problem is occurring and quickly isolate it.
In the absence of appropriate tools, automated visibility into problems on that level basically isn't possible. Instead, you need to convene a meeting involving representatives from each discipline, try to figure out who did what, and then extrapolate the solution to problem from there... not very efficient, to say the least.
Another key capability that you get from a scalable, cross-disciplinary build management process is improved communication among team functions that are involved in building the product. Today there is quite a bit of manual communication in most build processes. When a developer checks in code, for instance, she may need to e-mail a build engineer or someone else to initiate a build process. If the build engineer has left work early for a dental appointment the build might not happen until the next morning. These kinds of manual controls all decrease final quality as well as increase time-to-market.
All the various communication points that need to occur in a manual build process -- related to development, source control, building, testing, packaging, and deploying -- create numerous potential gaps and time sinks that serve to delay releases. And as releases are delayed features are cut, testing is sometimes cut back -- all kinds of things might happen in service to a delivery date, and many of them add risk to the project and negatively impact quality.
A further aspect of communication is documentation. In the case of homegrown build processes, team members within each discipline might be manually creating numerous documents that are quickly outdated. An advantage of cross-project, automated build management is that it tracks and controls all the build and release activities from beginning to end; therefore, these activities can be made self-documenting. This is yet another example of how structuring the build process makes less work for the team overall, not more.
The build management system should also make it easier to specify and enforce various standards through automation because, in order to make a process more predictable you need to standardize a few things so they are known across all the functional areas that are involved in the process.
Naming conventions are a good example of this. To automate activities across disciplines you need to ensure that the areas involved (development, configuration management, testing, etc.) adhere to appropriate naming standards so that they have a common frame of reference for identifying and associating build inputs and outputs across their particular areas.
Another area where standards are applicable is best-practice reuse of processes. Large, distributed development organizations engaged in multiple, complex projects (e.g., SOAs) benefit tremendously from an ability to stipulate certain best-practice activities for build management. For example, you might want to mandate that each project team capture specific build/deploy data in the form of reports that are relevant for compliance or internal governance purposes. When business-critical activities like these are automated, documented, and available for reuse, teams are much more likely to follow them. At the same time, empowering teams to reuse best practices rather than reinventing the wheel time and again on different projects helps you reduce project setup time, reduce rework, and enhance both efficiency and overall product quality.
Cross-disciplinary security is yet another advantage of a well-structured build and release management process. For example, certain roles or people might benefit from access to different elements of the software under construction at various points in the development lifecycle; to sanity-check some code by running a few tests, let's say. But they might not necessarily need to own or control those elements (e.g., the test scripts or whatever) at those points.
Being able to establish security in the form of authorizations and controls that go beyond each function's tool sets to wrap around the integration points in the build/deploy process can benefit both quality and efficiency. This is what can enable a developer to safely run a build without needing to involve a build engineer on a Sunday afternoon. The right controls help eliminate much of the waste and slack remaining in development processes, and actually increase flexibility.
Like security, status information is another aspect of software development that transcends any one discipline. When you're working to get a release out the door, how easy is it for you to determine project status in real-time? Are you in a testing phase or a packaging phase? Is the build passing all the tests, all but a few tests, etc.? When you lack top-down visibility on your process these kinds of questions can be challenging and time-consuming to answer. Enterprise-class build management systems can help larger teams address these kinds of cross-discipline issues.
Metrics are an aspect of the ability to ascertain project status. When everybody on the project works in the same building, status information is easier to come by. But when teams are spread out geographically, ad hoc communication does not suffice. You need to somehow measure key metrics across the distributed process to ascertain status.
What you really need, in fact, is an integrated view across and down into all your operational development activities from one central location, from which you can see and measure whatever you need to know about. Are you spending too much time in development or in testing; and, if so, why? When there is appropriate structure connecting the disciplines it becomes much easier to make these kinds of determinations.
By the same token, without high-level visibility into your entire process you really don't know at a glance where the kinks are, let alone how to work them out. Given appropriate metrics you can analyze process trends over time to identify bottlenecks and continuously improve your process. Absent metrics, you have to extrapolate those insights manually, using so-called "tribal knowledge," word-of-mouth opinions, dozens of e-mail, and so forth.
You can find a lot of different opinions about what constitutes "agility" or "Agile development" -- but you won't find a development manager who is going to argue with tools and processes that deliver better quality products to market faster.
More and more organizations are beginning to deploy Agile type practices that improve quality and reduce the number of defects through more frequent code integration. When developers integrate their code more frequently, they discover problems sooner and fix them before they move downstream. Unfortunately, for all but small development teams the benefits of Agile have gone mostly untapped, or are mostly confined to the development discipline. Development teams may be iterating more frequently, but without automated support that embodies appropriate structure and controls, build activities cannot be similarly accelerated, and so neither can test activities, etc.
Automating build management is one way to bring Agile to larger teams -- while also expanding their scope and value. In effect, enterprise-scale build automation has the potential to redefine what "Agile" is all about, because it allows Agile-driven efficiency gains to propagate beyond the development function to the build, test, package, and deploy functions. That is, while developers integrate code more frequently, robust build automation enables the other disciplines to do likewise: build more frequently, test more frequently, etc. By eliminating bottlenecks in the build/deploy cycle the whole software development lifecycle can flow with greater efficiency.
When you impose the right kind of cross-functional structure on your development process you can make it significantly more efficient. Say you're on a build management team in the US working with a development team in India, which has just checked in some source code. Do they now need to wait hours for you to wake up and build it? If so, that's not particularly "Agile."
When build management is properly automated, however, the offshore team could login to your build system, run the build process themselves (without you having to baby-sit them), get the results back, learn whether the code they checked in worked or not, fix various problems on the fly, and so on. By the time you come to work your colleagues might have gone through several such iterations before you are even aware of it.
Here's a real-world example: At one of our IBM Rational Build Forge customers, a very large and distributed development organization went from doing eighteen builds a week to doing 360 builds per week, propagating the quality and efficiency benefits of more frequent code integration out from the developers to the QA team (who can now test more builds more frequently) and across the whole software development lifecycle. Now that is "Agile."
To illustrate another way that robust build management can help make developers -- and the entire team -- more efficient, let's envision another typical software development scenario. Developers work within their local workspaces, coding and compiling. At some point each developer must check his or her code into the source control system to see whether it will work with all the code others have written and checked in. Often, code that works in a local workspace turns out to break (or break some other code) at build time -- an everyday cause of delays on most projects.
Part of the reason teams tend to work this way is that traditionally there has been a bit of a wall between development and configuration management, because these functions naturally value different things: speed versus process control, respectively. Developers resist constraints on their tools and process; therefore, whoever is managing the source code needs to keep the developers at a safe distance from it.
What if, instead, you could safely let developers play in the build management sandbox without compromising speed, flexibility, or process control on anyone's part? Imagine if your developers could pre-check their code by running private "integration builds" outside their local workspaces, but without checking their code in.
Think of how many would-be bugs would never make it into builds in the first place, and how many more builds would therefore pass, if developers could move beyond unit testing to further test and validate their new code outside the formal build process and without involving other developers' new code. Think of how much more frequently and confidently developers could check in their code -- thus further enhancing overall team efficiency.
These are key aspect of the vision for "developer self-service" and "continuous integration" that automated build management brings into reality. This higher level of efficiency and control empowers all the development disciplines to iterate more frequently, and improves productivity and quality across the board.
IBM Rational Build Forge provides an automated build management and software delivery process that scales to meet the needs of large development teams. Build Forge reduces manual activity at the vital integration points among disciplines, while automatically capturing relevant data for compliance, governance, or project management purposes.
Among its key features, Build Forge offers:
- A Web-based console for centralized management of silo'd build/release processes, as well as simplified user administration
- The ability to automatically correlate data from disparate source control, testing, and defect tracking tools to provide a high-level view of product development status
- Flexible integration with existing tools across multiple platforms so individual disciplines aren't constrained in their choice of best-of-breed solutions
- Support for threading so you can break down builds into discrete, independent processes that can run concurrently, optimizing hardware utilization and build times
- Auto-documentation of a Bill of Materials (a list of content and changes) for better reproducibility of builds, faster problem resolution, simplified compliance management, and improved testing accuracy
- A library of "best practice processes" that you can reuse from project to project to reduce setup time, improve scalability and enhance both efficiency and quality
- Auto-documentation of key processes as they evolve over time -- providing an audit trail of process changes for compliance management, making it easier to share and shift workloads, and reducing the risk of "brain drain" because in-depth knowledge of (arcane) processes is not essential to manage builds
- An open command line interface that lets teams create "intelligent links" among processes to optimize efficiency and improve troubleshooting
- Developers self-service capabilities; i.e., developers can pre-test their code (that is, give it a "trial run") before checking it in, by running builds safely on-demand from within an Eclipse-based IDE
Rational Build Forge enables teams to generate builds faster, more frequently, and with greater confidence, thus accelerating the whole development effort and supporting quality management initiatives.
While many software development disciplines (source control, defect tracking, testing, etc.) enjoy mature, robust automation, the ability for enterprise teams to automate the build and release management processes has only recently emerged. Meanwhile, accelerating trends like SOAs, escalating compliance risk, and the growing ubiquity of distributed development has put intense pressure on today's homegrown build/deploy systems.
The inability to produce consistent, accurate, and repeatable software builds has become a major gating factor constraining both quality and efficiency. In particular, silo'd, discipline-centric build processes make it difficult to share information across the software lifecycle, resulting in unnecessary errors, debilitating delays, and increased risk to cost and schedule.
Best practices and process improvement throughout the software development lifecycle has always been IBM Rational's core competency. The scalable build management infrastructure we offer in Rational Build Forge can extend quality management across the entire software development lifecycle, while improving team efficiency and productivity at the same time. A more robust build/release process that eliminates manual handoffs and improves visibility into project status means less time spent recovering from failures; and an improved ability to proactively pinpoint and resolve errors upstream before they impact builds and delay testing.
As enterprise teams gain experience with build and release automation such as that offered by Build Forge, they are likely to enjoy further process improvements driven by compressed development schedules and the ability to leverage long-term data from the build system to streamline activities, improve planning, and simplify both regulatory compliance and internal governance. Organizations can also strengthen their customer relationships through an enhanced ability to provide high-quality software in a timely manner on a consistent basis.
All these process improvements lead to an improved ability to manage software quality, resulting in optimal cost-efficiency, optimal time-to-value, and improved customer satisfaction. For many mid- to large-sized development teams or distributed teams the question is not whether you will implement scalable build management automation, but when.
The IBM Rational Build Forge Web page: http://www.ibm.com/software/awdtools/buildforge/index.html
The IBM Rational quality management Web page: http://www.ibm.com/software/rational/offerings/testing.html
White papers on Build Forge: http://www.ibm.com/developerworks/rational/library/06/0801_bfwhitepapers/
Peter Schuh's recent DeveloperWorks article on "Agile Configuration Management for Large Organizations": http://www.ibm.com/developerworks/rational/library/mar07/schuh/index.html
Dear Reader: A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
Carey Benge joined Build Forge in 2005 as Manager of Technical Sales. With more than twenty years in the information systems and technology sector for industry-leading companies, he has extensive experience applying technology toward solving industry-specific business problems. Prior to Build Forge he spent five years at Motive Communications, where he built and led Motive's world-wide technical sales organization. From 1994-1999 he was Director of Technical Marketing at IBM Tivoli through its explosive initial public offering and subsequent growth. He holds B.S. and M.S. degrees in Computer Science from Texas A&M University-Commerce.
Comments (Undergoing maintenance)
Table of contents
Local resources
My developerWorks community
Tags
Use the slider bar to see more or fewer tags.
Popular tags shows the top tags for this particular content zone (for example, Java technology, Linux, WebSphere).
My tags shows your tags for this particular content zone (for example, Java technology, Linux, WebSphere).
Popular article tags |
My article tagsSkip to tags list
Popular article tags |
My article tags
