Open source C/C++ unit testing tools, Part 3: Get to know CppTest

In this third and final article in the series on open source unit testing utilities, get to know CppTest, a simple and easy-to-use framework for developing unit tests.

Arpan Sen (arpansen@gmail.com), Independent author

Arpan Sen is a lead engineer working on the development of software in the electronic design automation industry. He has worked on several flavors of UNIX, including Solaris, SunOS, HP-UX, and IRIX as well as Linux and Microsoft Windows for several years. He takes a keen interest in software performance-optimization techniques, graph theory, and parallel computing. Arpan holds a post-graduate degree in software systems. You can reach him at arpansen@gmail.com.


developerWorks Contributing author
        level

23 February 2010

Also available in Chinese Russian

This article, the last in a series on open source tools for unit testing, takes a detailed look into CppTest, a powerful framework by Niklas Lundell. CppTest's greatest asset is that it's simple to understand and easy to adapt and use. Learn how to create unit tests and test suites using CppTest, test fixture design, and customize regression log formatting while becoming familiar with several useful CppTest-provided macros. If you're a more advanced user, this article also provides a comparison between the CppUnit and CppTest frameworks.

Frequently used acronyms

  • HTML: Hypertext Markup Language
  • I/O: Input/output
  • XML: Extensible Markup Language

Installation and usage

CppTest is available as a free download from Sourceforge (see Resources) under the GNU Lesser General Public License (GPL). Building the sources follows the usual open source configure-make format. The result is a static library named libcpptest. The client code must include the header file cppTest.h, which is part of the downloaded sources, as well as a link against the static library libcpptest.a. This article is based on CppTest version 1.1.0.


What is a test suite?

Unit testing is meant to test specific sections of the source code. In its simplest form, this testing involves a collection of C/C++ functions testing other C/C++ code. CppTest defines a class called Suite within the Test namespace that provides the basic test suite functions. User-defined test suites are required to extend this functionality by defining functions that work as actual unit tests. Listing 1 defines a class named myTest that has two functions, each of which tests a piece of source code. TEST_ADD is a macro meant to register the tests.

Listing 1. Extending the basic Test::Suite class
#include “cppTest.h”

class myTest : public Test::Suite { 
  void function1_to_test_some_code( );
  void function2_to_test_some_code( );

  myTest( ) { 
      TEST_ADD (myTest::function1_to_test_some_code); 
      TEST_ADD (myTest::function2_to_test_some_code); 
  } 
};

Extending the test suite

You can easily extend the test suite's functionality to create a hierarchy of test suites. The need for doing something like this stems from the fact that each test suite might test some specific code area—for example, a test suite for parsing, code generation, code optimization—for a compiler, and the hierarchy makes for better management over time. Listing 2 shows how you can create such a hierarchy.

Listing 2. Creating a unit test hierarchy
#include “cppTest.h”

class unitTestSuite1 : public Test::Suite { … } 
class unitTestSuite2 : public Test::Suite { … } 

class myTest : public Test::Suite { 
  myTest( ) { 
      add (std::auto_ptr<Test::Suite>(new unitTestSuite1( ))); 
      add (std::auto_ptr<Test::Suite>(new unitTestSuite2( ))); 
  } 
};

The add method belongs to the Suite class. Listing 3 shows its prototype (sourced from the header cpptest-suite.h).

Listing 3. The Suite::add method declaration
class Suite
{
 public:	
    …
    void add(std::auto_ptr<Suite> suite);
    ...
} ;

Running the first test

The run method of the Suite class is responsible for the execution of the tests. Listing 4 shows the test run.

Listing 4. Running a test suite in verbose mode
#include “cppTest.h”

class myTest : public Test::Suite { 
  void function1_to_test_some_code( ) { … };
  void function2_to_test_some_code( ) { … };

  myTest( ) { 
      TEST_ADD (myTest::function1_to_test_some_code); 
      TEST_ADD (myTest::function2_to_test_some_code); 
  } 
}; 

int main ( ) 
{ 
  myTest tests; 
  Test::TextOutput output(Test::TextOutput::Verbose);
  return tests.run(output);
}

The run method returns a Boolean value that is set to True only if all the individual unit tests were successful. The argument to the run method is an object of type TextOutput. The TextOutput class handles printing the test log. By default, the log is dumped onto the screen.

In addition to verbose mode, there's terse mode. The difference between these two modes is that verbose mode prints line number/file name information for assertion fails in individual tests, while terse mode only provides a count of the tests that passed or failed.

Continuing after failure

So, what happens if a single test fails? The decision on whether to continue rests solely with the client code. The default behavior is to continue executing the other tests. Listing 5 shows the prototype for the run method.

Listing 5. Prototype for the run method
bool Test::Suite::run(	
    Output &   output,
    bool 	        cont_after_fail = true	 
);

The second argument to the run method needs to be False for situations in which the regression must exit after detection of the first fail. It might not be obvious when the second flag should be set to False, however. Assume that the client code is trying to write information to a disk that is 100% full: The code will fail, and all further tests in that suite that have similar behavior will fail, too. It makes sense to stop regressions immediately in this situation.


Understanding output formatters

The idea behind output formatters is that you may need to have the regression run report in different formats: text, HTML, and so on. Hence, the run method itself does not dump the results, but accepts an object of type Output, which is responsible for displaying the results. Three different kinds of output formatters are available in CppTest:

  • Test::TextOutput. This is the simplest of all output handlers. The display mode could be either verbose or terse.
  • Test::CompilerOutput. The output is generated in a manner resembling a compiler build log.
  • Test::HtmlOutput. Fancy HTML output.

By default, all three formatters dump their output on std::cout. The constructors for the first two formatters accept an argument of type std::ostream, which is indicative of where the output should be dumped—in a file for future perusal, for example. You may also choose to create a customized version of the output formatter. To do so, the only requirement is that the user-defined formatter be derived from Test::Output. To understand the different output formats, consider the code in Listing 6.

Listing 6. Running the TEST_FAIL macro in terse mode
#include “cppTest.h”

class failTest1 : public Test::Suite { 
void always_fail( ) { 
  TEST_FAIL (“This always fails!\n”); 
} 
public: 
  failTest1( ) { TEST_ADD(failTest1::always_fail); } 
}; 

int main ( ) { 
  failTest1 test1;
  Test::TextOutput output(Test::TextOutput::Terse);
  return test1.run(output) ? 1 : 0;
}

Note that TEST_FAIL is a predefined macro in cppTest.h that causes an assertion fail. (More on this later.) Listing 7 shows the output.

Listing 7. Terse output showing only the fail count
failTest1: 1/1, 0% correct in 0.000000 seconds
Total: 1 tests, 0% correct in 0.000000 seconds

Listing 8 shows the output when you run the same code in the verbose mode.

Listing 8. Verbose output showing file/line information, the message, test suite information, and so on
failTest1: 1/1, 0% correct in 0.000000 seconds
        Test:    always_fail
        Suite:   failTest1
        File:     /home/arpan/test/mytest.cpp
        Line:    5
        Message: "This always fails!\n"

Total: 1 tests, 0% correct in 0.000000 seconds

Next, the code for using the compiler style formatting is shown in Listing 9.

Listing 9. Running the TEST_FAIL macro with compiler-style output formatting
#include “cppTest.h”

class failTest1 : public Test::Suite { 
void always_fail( ) { 
  TEST_FAIL (“This always fails!\n”); 
} 
public: 
  failTest1( ) { TEST_ADD(failTest1::always_fail); } 
}; 

int main ( ) { 
  failTest1 test1;
  Test::CompilerOutput output;
  return test1.run(output) ? 1 : 0;
}

Note the similarity in syntax with the GNU Compiler Collection (GCC)-generated compile log shown in Listing 10.

Listing 10. Terse output showing only the fail count
/home/arpan/test/mytest.cpp:5: “This always fails!\n”

By default, the compiler-formatted output is the GCC-style build log. However, it is possible to get the output in Microsoft® Visual C++® and Borland compiler formats, too. Listing 11 generates the Visual C++-style log that is dumped into an output file.

Listing 11. Running the TEST_FAIL macro with compiler-style output formatting
#include <ostream>

int main ( ) { 
  failTest1 test1;
  std::ofstream ofile;
  ofile.open("test.log");
  Test::CompilerOutput output(
      Test::CompilerOutput::MSVC, ofile);
 return test1.run(output) ? 1 : 0;
}

Listing 12 shows the content of the file test.log after the execution of the code in Listing 11.

Listing 12. Virtual C++-style compiler output
/home/arpan/test/mytest.cpp (5) :  “This always fails!\n”

Finally, here goes the fanciest of them all: the code for HtmlOutput usage. Note that the HTML formatter does not accept a file handle in the constructor, but instead relies on the generate method. The first argument to the generate method is an object of type std::ostream with default value std::cout (see the source header file cpptest-htmloutput.h for details). You may use a file handle to redirect the log elsewhere. Listing 13 provides an example.

Listing 13. HTML-style formatting
#include *<ostream>

int main ( ) { 
  failTest1 test1;
  std::ofstream ofile;
  ofile.open("test.log");
  Test::HtmlOutput output( );
  test1.run(output); 
  output.generate(ofile);
  return 0;
}

Listing 14 shows part of the HTML output generated in test.log.

Listing 14. Snippet from the generated HTML output
…
<table summary="Test Failure" class="table_result">
  <tr>
    <td style="width:15%" class="tablecell_title">Test</td>
    <td class="tablecell_success">failTest1::always_fail</td>
  </tr>
  <tr>
    <td style="width:15%" class="tablecell_title">File</td>
    <td class="tablecell_success">/home/arpan/test/mytest.cpp:18</td>
  </tr>
  <tr>
    <td style="width:15%" class="tablecell_title">Message</td>
    <td class="tablecell_success">"This always fails!\n"</td>
  </tr>
</table>
…

All about test fixtures

Unit tests that form part of the same test suite often have the same set of initialization requirements: You must create objects with certain parameters, open file handles/operating system ports, and so on. Instead of repeating the same code in each class method, a better way of approaching this issue is to use some common initialization and exit routines that are called for each test. You must define the setup and tear-down methods as part of his test suite. Listing 15 defines the test suite myTestWithFixtures, which uses fixtures.

Listing 15. Creating a test suite with fixtures
#include “cppTest.h”

class myTestWithFixtures : public Test::Suite { 
  void function1_to_test_some_code( );
  void function2_to_test_some_code( );

  public: 
  myTest( ) { 
      TEST_ADD (function1_to_test_some_code); 
      TEST_ADD (function2_to_test_some_code); 
  } 

  protected: 
    virtual void setup( ) { … };
    virtual void tear_down( ) { … };
};

Note that you don't need to make any explicit calls to the setup and tear-down methods. These routines don't need to be declared virtual unless you have plans to extend the test suite at a later time. Both routines must be of return type void and accept no arguments.


Learning the macros in CppTest

CppTest provides several useful macros that you use in the actual methods that test the client source code. These macros are internally defined in cpptest-assert.h, which cpptest.h includes. Some of the macros and their potential use cases are described next. Note that unless otherwise stated, the output shown is provided using verbose mode.

TEST_FAIL (message)

This macro, shown in Listing 16, is meant to imply unconditional failure. A typical situation in which you could use this macro is handling the results of a client function. If the result does not tally with any of the expected results, then an exception along with the message is thrown. When the TEST_FAIL macro gets hit, no further code in that specific unit test is executed.

Listing 16. Client code using the TEST_FAIL macro
void myTestSuite::unitTest1 ( ) { 
    int result = usercode( );
    switch (result) { 
       case 0: // Do Something 
       case 1: // Do Something 
       …
       default: TEST_FAIL (“Invalid result\n”); 
   }
}

TEST_ASSERT (expression)

This macro is similar to the C assert library routine, except that TEST_ASSERT works both in debug and release builds. If the expression is verified to be False, then an error is flagged. Listing 17 shows the internal implementation for this macro.

Listing 17. TEST_ASSERT macro implementation
#define TEST_ASSERT(expr)  \
{  	\
  if (!(expr))  \
  { \
  assertment(::Test::Source(__FILE__, __LINE__, #expr));	\
     if (!continue_after_failure()) return;  \
  } \
}

TEST_ASSERT_MSG (expression, message)

This macro is similar to TEST_ASSERT except that when the assertion fails, the message shows up in lieu of the expression in the output. Here's an assertion with and without the message.

TEST_ASSERT (1 + 1 == 0);
TEST_ASSERT (1 + 1 == 0, “Invalid expression”);

Listing 18 shows the output when this assertion is hit.

Listing 18. Output from the TEST_ASSERT and TEST_ASSERT_MSG macros
Test:    compare
Suite:   CompareTestSuite
File:     /home/arpan/test/mytest.cpp
Line:    91
Message: 1 + 1 == 0

Test:    compare
Suite:   CompareTestSuite
File:     /home/arpan/test/mytest.cpp
Line:    92
Message: Invalid Expression

TEST_ASSERT_DELTA (expression1, expression2, delta)

An exception is thrown if the difference between expression1 and expression2 exceeds delta. This macro is particularly useful when expression1 and expression2 are floating-point numbers—for example, depending on how the rounding off is actually done, 4.3 may be stored as 4.299999 or 4.300001, and as such, a delta is needed for the comparison to work. Yet another example is testing the code for operating system I/O: The time it takes to open a file can't be the same every time, but it has to be within a certain range.

TEST_ASSERT_DELTA_MSG (expression, message)

This macro is similar to the TEST_ASSERT_DELTA macro, except that when the assertion fails, a message is also issued.

TEST_THROWS (expression, exception)

This macro verifies the expression and expects an exception. An assertion is triggered if no exception is caught. Note that the actual value of the expression is not put to any test: It is the exception that is being tested. Consider the code in Listing 19.

Listing 19. Handling integer exceptions
class myTest1 : public Test::Suite { 
  …
  void func1( ) { 
     TEST_THROWS (userCode( ), int);
  }
  public: 
     myTest1( ) { TEST_ADD(myTest1::func1); } 
}; 

void userCode( ) throws(int) { 
   …
   throw int;
}

Note that the return type of the userCode routine is immaterial: It might as well be a double or an integer. Because userCode is throwing an exception of type int unconditionally here, the test will pass fine.

TEST_THROWS_ANYTHING (expression)

Sometimes, the client routine throws different kinds of exceptions depending on the situation. To handle this situation, you have the TEST_THROWS_ANYTHING macro, which does not specify an expected exception type. As long as some exception is caught after execution of the client code, the assertion will not be fired.

TEST_THROWS_MSG (expression, exception, message)

This macro is similar to TEST_THROWS, except that in this case, the message is printed, not the expression. Consider the following code:

TEST_THROWS(userCode( ), int);
TEST_THROWS(userCode( ), int, “No expected exception of type int”);

Listing 20 displays the output when these assertions fail.

Listing 20. Output from the TEST_THROWS and TEST_THROWS_MSG macros
Test:    func1
Suite:   myTest1
File:    /home/arpan/test/mytest.cpp
Line:    24
Message: userCode()

Test:    func2
Suite:   myTest1
File:    /home/arpan/test/mytest.cpp
Line:    32
Message: No expected exception of type int

Comparison between CppUnit and CppTest

Part 2 of this series dealt with CppUnit, the other popular choice of open source unit testing frameworks. CppTest is a much simpler framework than CppUnit, but it gets the job done. Here's a quick comparison of how these two powerful tools match up against one another:

  • Ease of creating a unit test and test suite. Both CppUnit and CppTest create unit tests of class methods, with the class itself derived from some tool-provided Test class. The syntax for CppTest is slightly simpler, though, with the test registration happening inside the class constructor. In the case of CppUnit, the additional macros CPPUNIT_TEST_SUITE and CPPUNIT_TEST_SUITE_ENDS are needed.
  • Running the tests.CppTest simply calls the run method on the test suite, while CppUnit uses a separate TestRunner class whose run method is invoked for running the tests.
  • Extending the testing hierarchy. In the case of CppTest, it is always possible to extend the previous test suite by creating a new class that inherits from the old one. The new class will define some additional functions that add to the unit test pool. You simply invoke the run method on the object of the new class type. CppUnit, in contrast, requires that you use the macro CPPUNIT_TEST_SUB_SUITE along with class inheritance to achieve the same effect.
  • Generating formatted output. Both CppTest and CppUnit have the ability to customize the output. However, although CppTest has a useful, predefined HTML output formatter, CppUnit does not. However, CppUnit exclusively supports XML formatting. Both support text and compiler style formats.
  • Creating test fixtures. To use test fixtures, CppUnit requires that the test class be derived from CppUnit::TestFixture. You must provide definitions for the setup and tear-down routines. In the case of CppTest, you need to provide definitions only for the setup and tear-down routines. This is definitely a better solution, as it keeps the client code simple.
  • Predefined utility macro support. Both CppTest and CppUnit have a comparable set of macros for asserts, handling floats, and so on.
  • Header files.CppTest requires that you include a single header file, while CppUnit client code must include multiple headers, like HelperMacros.h and TextTestRunner.h, depending on the features used.

Conclusion

Unit testing is fundamental to how you develop today's software, and CppTest is yet another tool in the C/C++ developer's arsenal to tackle code testing, thereby avoiding maintenance hassles. It is interesting to note that all three tools covered in this three-part series—the Boost testing framework, CppUnit, and CppTest—use the same basic concepts, such as fixtures, macros for assertions, and output formatters. Each tool being open source, you could easily further customize the code for your needs (for example, an XML outputter for CppTest). What are you waiting for?

Resources

Learn

Get products and technologies

Discuss

Comments

developerWorks: Sign in

Required fields are indicated with an asterisk (*).


Need an IBM ID?
Forgot your IBM ID?


Forgot your password?
Change your password

By clicking Submit, you agree to the developerWorks terms of use.

 


The first time you sign into developerWorks, a profile is created for you. Information in your profile (your name, country/region, and company name) is displayed to the public and will accompany any content you post, unless you opt to hide your company name. You may update your IBM account at any time.

All information submitted is secure.

Choose your display name



The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the content you post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your email address for privacy reasons.

Required fields are indicated with an asterisk (*).

(Must be between 3 – 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

 


All information submitted is secure.

Dig deeper into AIX and Unix on developerWorks


static.content.url=http://www.ibm.com/developerworks/js/artrating/
SITE_ID=1
Zone=AIX and UNIX
ArticleID=468766
ArticleTitle=Open source C/C++ unit testing tools, Part 3: Get to know CppTest
publish-date=02232010