People always look for the next big thing that will change our industry. My aspirations are usually a bit less grand. I look for the next incremental or evolutionary step that moves us forward. There have certainly been some grand ideas that have fundamentally changed the way we create our software systems -- like incremental, iterative development -- but, in general, none of them has completely changed our view of the universe.
Lately I've gotten interested in an area that I think has the potential to create the next fundamental change in software development: collaboration and creativity support. The agile movement brought into the spotlight the need for teams to communicate continuously and well. It emphasized other practices that good developers have always known, and codified them into a growing body of practices and principles. Agile methods are but one way to develop software. The common feature of all successful projects I've encountered has been effective communication and collaboration among the whole team.
The whole team includes the customers, developers, managers, quality professionals, writers, and everyone else who is involved with the project. This has been true regardless of whether the project uses a waterfall, iterative, or agile methodology.
Many tools claim to support collaboration, while the actual degree of collaboration support, until recently, has been minimal. However, a new breed of tools has begun to appear that gives me hope that we're going to see a dramatic increase in real support for team collaboration. Following on the heels of collaboration support is creativity support. These two areas -- collaboration and creativity -- are closely related. This month I'll look at collaboration, some of the tools that are now available, and look ahead to where we might be going with the concept. Success in developing automatic support for collaboration and creativity might take us to the next level in our ability to produce great software systems.
Understanding collaboration and creativity
Let's start by making sure that we have a common understanding of collaboration and creativity. Although there's a great deal written about creativity, collaboration is the easier of the two ideas to grasp, because we know more about it in the context of software development. I define collaboration for the purposes of this article to be a group of people working together to attain a common goal. Collaboration, by its very nature, means that there is more than one person involved. Collaboration is also goal-oriented. People collaborate for a specific purpose. In software development that purpose is the successful development and deployment of a software system on time and within budget.
Collaboration requires communication, but communication alone is not sufficient. Collaboration requires agreement on the goals of the collaborative effort, an understanding of how the goals will be attained, and an awareness of the state of the collaborative effort. I'll look at this in more depth below.
I said we know more about collaboration than we do about creativity, but I'm not sure that collaboration has really been studied in a scientific sense as much as creativity. I simply mean that we intuitively understand collaboration. Anyone who has worked or played as part of a team understands how collaboration works. As I've searched for research papers on collaboration and creativity, the creativity papers seem more numerous. The two are strongly connected, however, and they share some of the same characteristics from the viewpoint of understanding the nature of each.
Creativity has been the focus of several studies and research efforts in various disciplines. Psychologists try to understand the nature of creativity. How do creative people come up with their ideas? What is the working style of creative people? Can we teach people to be creative? Social scientists, computer scientists, and others have joined in this exploration because it is important. As globalization has become the norm rather than the exception, some countries will thrive because they can deliver services cheaper than others and some will thrive because they are creative forces. In the future I believe we'll see the need for productivity, cost of services and goods, and creativity needed in some mixture to keep an economy competitive. This is a great challenge and the stakes are very high.
Among the papers I've read lately on this topic are those by Mihaly Csikszenthmihalyi.1 Csikszenthmihalyi says there are three components to understanding creativity:
- Domain: The domain consists of a set of symbols rules and procedures, such as the domain of mathematics or physics.
- Field: The field is the set of people who act as "gatekeepers" to the domain. These are the elder statesmen of the domain who decide if something is worthy to become part of the domain.
- Individual: The individual component of creativity is "when a person... has a new idea or sees a new pattern, and when this novelty is selected by the appropriate field for inclusion in the relevant domain."
Researchers use this framework to guide their work on understanding creativity and how to support it. I will follow up on this below.
Relationship between collaboration and creativity
How are these two concepts, collaboration and creativity, related? My view of the relationship is shown in Figure 1. By definition collaboration involves a team, and collaborative efforts are task-based, but there is often some bit of exploration, such as the "spike" in eXtreme Programming.2 I'm not sure that one would call this work creative in light of the components given by Csikszenthmihalyi. The creative effort can be both individual and team-based, but the work is mostly exploratory.
Figure 1: Relationship between collaboration and creativity
In terms of software development, most software applications are written to solve a particular problem. So the typical IT project is task-based and operates in the upper half of Figure 1. Projects that develop software tools and others that have similar profiles, however, work in a different way. Typically an individual or small group invests creative effort to develop the core ideas behind the tool. Once various approaches have been explored and one settled upon, the project shifts to a collaborative mode where a team builds the product.3
Collaborative efforts succeed because of several contributing factors. Some of the most important are:
- Clear identification of the tasks and the team members assigned to complete the tasks
- Awareness of the team members' activities and progress and the overall progress of the project towards completion
- Rich history that offers views of the state of the project at any time in the past
- Effective synchronous and asynchronous communication
A history of collaboration tools
Supporting successful collaborative efforts should support these factors. Let's look at how tools support them. I'll start with the simplest and oldest and work forward to where we are today.
Helping teams identify tasks and task assignments has been a part of project management tools for decades. On very simple projects one can use a spreadsheet with a few macros. Projects that have a large team, multiple components that must be integrated, and a long development cycle use sophisticated, complex project scheduling tools.
Project management tools support collaboration somewhat, but one would not categorize them necessarily as a collaboration support tool. One reason for this is that they don't make the team members aware of their teammates' activities easily. If you're a developer and your project uses a sophisticated project management tool, ask yourself when you last used that tool to find out what another developer was doing? Quite possibly the answer is "never."
The issue of awareness is an important one. A good collaboration tool will make it easy for team members to be aware of what their fellow team members are doing. The tool, or tools, will also make information about the overall progress on the project available to everyone on the team with little or no effort. It's a matter of productivity. If it takes time to find the information one needs, that's time that could be spent on a more primary task, such as coding the implementation of a new feature. Also, if it takes time and effort to locate the information, people won't access it regularly. They will only access it when a crisis arises. The information at this point, while critical, is used in a reactive mode.
Using project progress and task assignment information proactively makes more sense. The information has more value. Consider a programmer who's assigned to implement a new feature on an application. As she works on it, she realizes that another part of the system will use the feature she's implementing. She could just implement the feature and after she checks it in, the programmer who needs to use her feature will have to adapt his code. If she can quickly find out who is assigned to the work that will make use of her feature she can work with that programmer to implement the feature in such a way that it is best adapted to both their needs. This is what I mean by proactive use of the information.
Rich history information has many uses. With respect to collaboration it allows the team to review a project retrospectively and learn where they might have made better choices or to revert to a previous version of the project and pursue another path. Many of the capabilities for capturing rich history information have been around for years with version control systems. IBM® Rational® ClearCase® has been a premium entry in the software configuration management (SCM) tool market for a long time. The ability to branch, merge, and do parallel development went into the design of the product. Such products provide the basis for providing rich history, but we still need more.
The role of communication
We want to be able to go back in time to a point where we have a complete picture of the project. Of course this picture includes the artifacts that are archived in our version control systems, but it also includes information that is not so easily archived in typical version control systems. This information might consist of emails, Web meeting transcripts and recordings, and so on. The information must also be presented in ways that encourage exploration and collaboration.
This brings us to communication. We all know that it's usually more effective to talk to someone either in person or by phone than to write email. We're able to clarify issues and communicate intentions much better when we're conversing in real-time. In today's highly distributed environments, it's not always possible, and in fact it's becoming rarer that people are in the same location. We want to have communication that is as synchronous as possible whenever we can. But if we can't do that, we need to have effective asynchronous communication.
Communication produces valuable artifacts for development projects -- information that the team should use for collaborative efforts. One challenge we face is, how can we capture communication and make the information available to the team on an as-needed basis? I've recently been involved with several student projects at Worcester Polytechnic Institute (WPI) that have started to address this problem.
Collaboration support tools today
The above comments are quite general. I want to now look at the type of tools we have available to support collaboration today. This list is not exhaustive, but provides a reasonable sample of what's available to the practitioners today.
Online collaborative communities and project environments. The number of online communities where teams can initiate projects and collaborate has exploded. Some of these, such as sourceforge.net and the Java community projects are two examples of open source communities.4 Commercial products support the same capabilities and more for organizations that want to manage their own portfolio of projects. As long as the organization has a process in place for using these products they provide a great start for collaboration among distributed teams.
Version control and source code management systems. These are essential for any team, distributed or co-located. Most of the products in the previous category contain these products or provide interfaces to the products.
Bug-tracking tools. Although they provide asynchronous capabilities only, these tools are another essential requirement for development teams. Such tools provide awareness of project health (the number of defects) and task assignment information.
Web meeting tools. Several Web meeting tools were developed as the Internet and network technologies developed into a mature, fast communications medium. One hears commercials on radio and television for such tools and reads advertisements for them in many popular magazines. The tools vary in their capabilities, but their essence is to allow a group to conduct a virtual meeting that enables video, audio, and shared computer desktop capabilities. The experiential quality afforded by such tools depends upon many factors, including the participants' willingness to accept the tools and learn to use them properly.
Collaboration capabilities embedded in IDEs. Many interactive development environments contain collaboration capabilities. The Eclipse platform has led the way with a number of plug-ins to make collaboration possible and easier to Eclipse users. One of the more mature projects and plug-ins are the Mylyn project5. Mylyn provides task-focused tools to a programmer's work. It also integrates with several bug-tracking tools. At WPI we have the Webfoot project6, which is an ongoing project to provide rich collaboration capabilities to Eclipse users. A nice technology framework for developing tools is the Eclipse Communication Framework7. The project has been working for a couple of years to provide the capabilities for developers to build their own tools. It comes with some example applications such as a real-time shared editor with the latest Eclipse release.
Other IDEs have added many of the capabilities that Eclipse includes. The NetBeans IDE has had a very nice shared editing capability for awhile. Other tools offer more focused capabilities, like Smart Bear's Code Collaborator.
Unfortunately, there are few standards for interoperability with these tools, which does limit the development teams' choices. The whole team must use the same IDE to reap the benefits of the collaboration capabilities.
Continuous integration tools. Continuous integration tools have become popular in the last few years. These work in conjunction with SCM systems to automatically watch the code repository and build the system (including running tests and packaging artifacts) periodically or whenever anything changes. They also have the ability to notify the whole team, or selected members of the team. They often provide a dashboard that makes the current build status immediately available.
Jazz: The next step
I'm really excited to see IBM Rational's Jazz platform make it to market.8 It puts a lot of the required capability for effective collaboration into one platform. And, it's built on the Eclipse platform so that you don't need to learn a lot of new tools. The Rational Team Concert (RTC) product is the first product within the Jazz line. I've been using early versions of it at WPI and have found it a joy to use. The setup and administration were simple and in my tests with a few of my students, it proved quite useful. I'm hoping to roll it out to my classes this year and have the student teams use it.
The three versions of RTC have an impressive list of features, shown in Figure 2. Compare this to the list of collaboration attributes that I think are necessary and you'll see a pretty good match.
What I particularly like about RTC is the awareness capability it provides, as well as the process customization potential. Now, team members can see what others are doing at any given time and, if a question arises and the person with the answer is online, the programmers can easily collaborate on getting the answer and ensure that they are in agreement on any issues. This is a huge step forward, in my opinion.
I haven't talked a lot about process in this article. However, every team has a process whether it's written down or not. RTC delivers a straightforward agile process out of the box. But the process is customizable without a lot of effort. I haven't done a lot of customization to the process yet, but I expect to have a complete, simple process for my students ready this year.
If you haven't taken a look at RTC, I urge you to check it out. There's an Express edition that is free for the first three users. That will give you a feel for what I'm talking about.
Figure 2: IBM Rational Team Concert features
We're seeing the beginning of a revolution in collaboration support tools. Jazz, while being quite impressive, is just the beginning. I look for a lot of software tool vendors to begin releasing additional collaboration capabilities. My hope is that some reasonable set of interoperability standards will result from the effort so that developers will be free to adopt the tools they're most comfortable and productive with and still be able to collaborate with team members who are using different tools.
I expect that support for creativity will be equally as important as collaboration support becomes more entrenched. I'll discuss creativity support in a later article.
- Csikszenthmihalyi's books on Flow have become popular best sellers, especially for managers who are looking to find ways to get their teams to develop creative solutions to problems that can be turned into commercial successes.
- The spike is a short period of time used to explore how to use new technologies or different ways of solving a problem.
- I'm ignoring those projects that are single-person from start to finish. More than a single person creates most interesting software.
- See http://www.sourceforge.net and http://community.java.net/projects/
References to Collaboration and Creativity
 L.T. Cheng, C. De Souza, S. Hupfer et al., "Building Collaboration into IDEs," ACM Queue, vol. 1, no. 9, pp. 11, 2004.
 M. Csikszentmihalyi, Flow: The Psychology of Optimal Experience, New York: HarperCollins, 1990.
 M. Csikszentmihalyi, Creativity: Flow and the Psychology of Discovery and Inventory, New York: HarperCollins, 1996.
 M. Csikszentmihalyi, Finding Flow: The Psychology of Finding Engagement with Everyday Life, New York: Basic Books, 1997.
 T. Hewett, M. Czerwinski, M. Terry et al., "Creativity Support Tool Evaluation Methods and Metrics," in Creativity Support Tools, Washington, D.C., 2005, pp. 16.
 B. Schneiderman, "Creativity Support Tools Accelerating Discovery and Innovation," Communications of the ACM, vol. 50, no. 12, pp. 13, 2007.
 B. Schneiderman, G. Fischer, M. Czerwinski et al., "Introduction to Workshop Report," in Creativity Support Tools, Washington, D.C., 2005, pp. 7.
 B. Schneiderman, and others, "Report of Workshop on Creativity Support Tools," http://www.cs.umd.edu/hcil/CST/Papers/creativitybook_final.pdf [July 15, 2005].
 B. Schneiderman, and others, "Creativity Support Tools: Report from a U.S. National Science Foundation Sponsored Workshop," International Journal of Human-Computer Interaction, vol. 20, no. 2, pp. 17, 2006.
- Participate in the discussion forum.
- A new forum has been created specifically for Rational Edge articles, so now you can share your thoughts about this or other articles in the current issue or our archives. Read what your colleagues the world over have to say, generate your own discussion, or join discussions in progress. Begin by clicking HERE.
- Global Rational User Group Community