In the standard build process, developing applications under UNIX with C/C++ generally involves using a compiler such as gcc with some kind of build tool, such as make. The problem with make and just about any C compiler is the way in which the C preprocessor and header files work. If you have a look at a typical C source file, it will incorporate a number of #include references to the various header files.

Each time you compile a file, the C preprocessor (cpp) parses and then includes each of these files and any files that are in turn referenced. By parsing the content, cpp can turn what was a fairly basic 1-KB source file into an 8-KB source file and, in the process, incorporate tens or even hundreds of header files. In a typical development project, a number of header files relevant to the project may be included multiple times in different source files, and each header file may itself reference many others.

During a typical build, the make tool simplifies the process considerably by compiling only the files that have changed since the target was last built. For example, the directory in Listing 1 shows that the foo.o object is older than the last modification time of the corresponding foo.c source file. Meanwhile, bar.o is newer than bar.c. With a suitably configured Makefile, only foo.o would be recompiled from its source.

The problem with make is that it uses the timestamp as the only method of actually determining whether the file needs to be recompiled. Because make also takes header files into the equation, a simple header file change will trigger a later timestamp and therefore a recompilation of the source files. But because of the way CPP works, for example, using the #ifdef statement, these changes may not affect the files that include them. A simple change of modification time can therefore mean that you end up re-compiling source files for no reason. Individually, that time can be significant. Across a team, the effects may be even more pronounced as multiple developers may end up repeating the process many times, maybe even simultaneously over the course of a typical day.

total 808
-rw-------  1 mc  mc    5123 24 Jul 14:17 bar.c
-rw-------  1 mc  mc   39474 24 Jul 14:19 bar.o
-rw-------  1 mc  mc    7856 24 Jul 14:17 foo.c
-rw-------  1 mc  mc   28443 24 Jul 14:19 foo.o
-rwx--x--x  1 mc  mc  319742 24 Jul 14:19 foobar*
-rw-------  1 mc  mc    1045 24 Jul 14:21 foobar.h

Back to top

Using ccache

The ccache (short for compiler cache) tool caches the object files generated by a compilation. It uses a hash based on the output of CPP to provide a reliable method of determining whether the raw source before actual compilation has been changed. This method is more reliable than the timestamp system used by make. For example, using make without ccache on the source in Listing 2 would compile the file as normal.

#include "foobar.h"

void main(void)
{
}

If we edit one of the header files referenced by foobar.h, then standard make would definitely recompile the file in Listing 2, irrespective of the actual changes. With ccache, the output of the CPP version of the file would be checked with the hash of the previous compilation. If they matched, nothing would be done. If they didn't, it would recompile. It's as simple as that. On a single file, you are not going to see much advantage, but for over 20 or 30 or, certainly, hundreds, of files, the speed gains would be significant.

Installation

Installing ccache and using it is not as complicated as you might think. It doesn't replace or in any way affect the way you already use your compiler; instead, it acts as an interface between you and your compiler, so you can choose whether or not to use it according to your needs. To install ccache, download the source directly from the Samba group or a local mirror (see Resources at end of this article). Unpack the contents of the file:

$ bunzip2 -c ccache-2.3.tar.bz2|tar xf -

Change into the directory:

$ cd ccache-2.3

Configure:

$ ./configure

Build:

$ make

And finally install ccache:

$ make install

You're all ready to go!

Deployment

As mentioned above, ccache works by sitting between you and your normal compiler. Instead of calling gcc, you call ccache with gcc as the first argument. For example, to compile a file from the command line, you would normally use:

$ gcc foo.c

To use ccache you would type:

$ ccache gcc foo.c

On a single compilation of a file like this, especially if it is the first time the file has been compiled using ccache, you won't see any benefit, because the compilation information has not been cached yet. Therefore, it is generally more effective to configure ccache as a permanent replacement for your main compiler. To do this, set the value of the CC environment variable:

$ export set CC='ccache gcc'

If you want to enable ccache only on a project basis, say when compiling third-party tools such as Perl, then you can either use the environment trick or tell the configure script or make command what C compiler to use.

Controlling caches

By default, ccache uses a directory within the current user's home directory ($HOME/.ccache) to hold the cache information. In a team environment, you'll want to use a central location for the cache so everybody can use the cache information during builds. Another environment variable, CCACHE_DIR, specifies the location of the cache directory. In a single-machine environment, set this to a directory that is accessible to everybody who needs it. For a greater speed boost, and providing you have the memory to support it, use a directory mounted using tmpfs. You could get an additional speed boost of between 10 and 25 percent.

If you are using ccache across a network of machines, make sure the directory you share is exported over NFS and mounted on each client. Again, you can use a tmpfs filesystem if you want to get an extra speed boost.

Some additional options give you further control over the cache settings:

• The CCACHE_LOGFILE environment variable defines the location of a log file that will be populated as you use ccache.
  
  
• Use the -s command line option with ccache to get statistics about the cache performance (see Listing 3).
  
  
• Use the -M command line option to set the maximum size of the cache. The default is 1GB. The settings for the cache are written to the cache directory, so you can have different cache sizes in different locations for different users and groups.
  
  
• The -F option sets the maximum number of files, rounded to the nearest 16, for the cache directory. As with -M it should only need to be used when you want to change the configuration.
  
  
• The -c option cleans the cache. You shouldn't normally need to use this, as ccache updates the information during execution, but if you reuse a cache directory that has not been used for a file, you might want to try this option.
  
  
• The -C option completely clears the cache.

cache hit                             44
cache miss                           152
called for link                      107
compile failed                        11
no input file                          2
files in cache                       304
cache size                           8.8 MB
max cache size                     976.6 MB

Once you've set the initial options and configured the directory and cache size you want, there's no need to change anything. There's also no need to perform any regular maintenance.

Back to top

Combining ccache and distcc

You may already be aware of distcc, another tool from the Samba group, which allows you to spread the compilation across a number of machines. This effectively increases the number of simultaneous compilations that can occur, just as if you had used the multiple jobs option with make (with the -j command line option). The distcc system works by using on each host a daemon that accepts source files in their final pre-parsed form, and then compiles the files locally before returning the resulting object file.

When used properly, you can generally expect to get a slightly less than linear decrease in build times for each new identical node you add, although you'll only see the effects on projects where you have more than one source file, as distcc only distributes whole source files.

Because distcc distributes files in their parsed state, you can combine ccache, which speeds up the C preprocessing portion, with distcc to perform the actual compilation into object code. To use distcc and ccache in this way, configure distcc on your hosts and distcc and ccache on your main development machine.

Now set up your environment variables on the machine on which you want to build the project, as shown in Listing 4.

export set DISTCC_HOSTS='localhost atuin nautilus pteppic kernel'
export set CCACHE_DIR=/Data/Cache/CCache
export set CCACHE_PREFIX=distcc
export set CCACHE_LOGFILE=/Data/Cache/CCache.log
export set CC='ccache gcc'

The variables are defined as follows:

• DISTCC_HOSTS specifies the hosts to distribute work to.
  
  
• CCACHE_DIR specifies the location of the ccache directory.
  
  
• CCACHE_PREFIX defines a prefix to use when ccache calls the true compiler to compile the source (after preprocessing).
  
  
• CC sets the name of the C compiler to use in the first instance (ccache).

Now when you run make with the -j option to specify the number of the simultaneous compiles to perform, the file will first be parsed with ccache (using the cache if necessary) before being distributed to one of the distcc hosts.

Although distcc speeds up the process of compiling, it doesn't alter the basic limitations of the environment. For example, you shouldn't set the number of simultaneous jobs operated by make to more than twice the number of available CPUs. For example, if you have four two-way machines, you won't notice much improvement in speed if you set the job value above 16.

Back to top

Statistics

Now that everything is set up, it is time to see how much of a difference it makes. I've run a series of tests here building Perl. We need something meaty to compile, because ccache works best when it has cached the parsed header files. This is just the make phase, which occurs after we've run a standard configure (using configure.gnu), and it includes all the stages, even those not related to compiling code. These non-compiler operations won't affect the overall statistics.

As previously mentioned, the effects of ccache won't be felt on the first compilation. It's the recompilation, where you are reusing previous preprocessor passes, that will make a difference. The recompile times in Table 1 are based on simply doing a touch on each of the C source files in the main Perl source directory. I've included the times for building with plain gcc, ccache+gcc, and ccache+distcc+gcc in a four-node network with various values of concurrent distcc jobs.

Table 1. The recompile times

Wow! Just using ccache alone saves almost 3 minutes (well, 2 minutes and 45 seconds) on the build time of Perl, all because ccache has kept pre-parsed versions of the header files and used them in place of continually re-running cpp on each source file. Incorporating distcc into the process gives results in an overall speed increase as well as slightly faster recompile times.

Back to top

Summary

In this article, you've seen how much of a speed improvement you can obtain from a relatively straightforward and easy-to-use tool such as ccache. You can improve compile times even more when using ccache in combination with distcc. And when using these tools in a team environment, you could save hours every day just in compile times. That means fewer excuses for coffee breaks for your development staff, but also a decrease in development times for your applications.

Resources

• Download ccache and distcc from Samba.org.
  
  
• If you need help compiling software such as ccache and distcc from sources, check out the Compiling and installing software from sources tutorial (developerWorks, August 2000).
  
  
• For an introduction to the tmpfs filesystem, see Daniel Robbins' article Common threads: Advanced filesystem implementor's guide, Part 3: Using the virtual memory (VM) filesystem and bind mounts (developerWorks, September 2001).
  
  
• For a good introduction to using GCC on AIX on IBM eServer™ pSeries® machines, read Tips and tricks of using the GNU C/C++ compiler on AIX (developerWorks, October 2003).
  
  
• Find more resources for Linux developers in the developerWorks Linux zone.
  
  
• Browse for books on these and other technical topics.
  
  
• Develop and test your Linux applications using the latest IBM tools and middleware with a developerWorks Subscription: you get IBM software from WebSphere®, DB2®, Lotus®, Rational®, and Tivoli®, and a license to use the software for 12 months, all for less money than you might think.
  
  
• Download no-charge trial versions of selected developerWorks Subscription products that run on Linux, including WebSphere Studio Site Developer, WebSphere SDK for Web services, WebSphere Application Server, DB2 Universal Database Personal Developers Edition, Tivoli Access Manager, and Lotus Domino Server, from the Speed-start your Linux app section of developerWorks. For an even speedier start, help yourself to a product-by-product collection of how-to articles and tech support.
  
  

About the author

Rate this article

Comments

Back to top

Table of contents

• Using ccache
• Combining ccache and distcc
• Statistics
• Summary
• Resources
• About the author
• Comments