 | Level: Intermediate Bruce Tate (bruce@rapidred.com), CTO, RapidRed
14 Aug 2007 Testing is firmly entrenched in the Ruby on Rails community.
Many tools can help you, from the Rails stack to RCov for coverage to
Mocha and FlexMock for enhancing your test cases. But different tools
often support diverging strategies. Learn about the trade-offs of several basic testing strategies.
One of the most distinctive aspects of the Rails platform is the Ruby
language itself. As a dynamically typed language, Ruby has great
flexibility, convenience, and power, but at a cost. Dynamically typed
languages do not have a compiler that can catch certain kinds of errors,
including relatively common type errors and some misspellings. Very early
on, users of dynamically typed object-oriented languages learned that they
had to test.
The Ruby on Rails community embraces testing like the USA embraces
American Idol. They watch their test case results roll by with great
regularity. Ruby developers talk about testing; they blog about it; they
even participate behind the scenes, not with cell phone votes but by
contributing open source frameworks.
Without testing, Ruby applications would have a much higher incidence of
error per line of code than they do. With testing, you can have the
benefits of dynamically typed languages with much less of the downside. In
this article, I won't bore you with basic decisions, like whether
you should test at all, or how you can convince your managers that the testing
effort is worthwhile. I'll assume you already test. Instead, I'll break down some of the more subtle testing decisions that every
Ruby project lead must eventually make. I'll talk about how you can
measure your testing coverage, and how much testing you should do. I'll
walk you through the basic out-of-the-box testing techniques, and how those
compare with the newer mocking frameworks. Rather than providing an
exhaustive tutorial, I'll give you a few examples of the techniques we
used to build ChangingThePresent.org (see Resources) so you'll have a basic flavor of the
techniques in play. Then I'll break through the strengths and weaknesses
of the various techniques as we see them.
Rails testing, out of the box
The Rails framework has surprisingly robust testing before you add a
single gem. With minimal effort, you can specify a repeatable database
setup, send simulated HTTP messages to your Web applications, and do three
kinds of tests: unit tests, functional tests, and integration tests. The following section
provides a brief example of each.
Unit tests
Unit tests exercise Rails model code and sometimes helpers. Your unit
tests will make sure your model does what you've built it to do and that
the associations in your model behave as you think they should. As you
recall, Rails models are objects that wrap a single database table. For
the most part, each database column is an attribute on the model. Rails
helpers are functions that help to simplify model, view, or controller
code. You need to make sure that each model or helper has a test. At
ChangingThePresent, our unit tests for the most basic models are very
thin.
Listing 1: A basic model test
require File.dirname(__FILE__) + '/../test_helper'
class BannerStyleTest < Test::Unit::TestCase
fixtures :banner_styles
def test_associations
assert_working_associations
end
def test_validation_with_incorrect_specs_should_fail
bs = BannerStyle.new(:height => 10, :width => 10, :format => 'vertical_rectangle',
:model_name => 'Nonprofit')
assert !bs.save, bs.errors.inspect
bs2 = BannerStyle.new(:height => 400, :width => 240, :format => 'vertical_rectangle',
:model_name => 'Nonprofit')
assert bs2.save, bs2.errors.inspect
end
...
end
|
In Listing 1, you see a condensed test case with two tests. Banner styles create simple advertising banners. The sizes and shapes of each are based on a loose set of standards. The application uses a table of standards to ensure that any new banner conforms to the standards. The first test uses a helper to exercise all of the associations on BannerStyle through reflection, as in Listing 2. The second test makes sure that a banner with an incorrect height and width cannot be saved, and that the model correctly saves a banner with valid specs.
Listing 2. A helper to test working associations
def assert_working_associations(m=nil)
m ||= self.class.to_s.sub(/Test$/, '').constantize
@m = m.new
m.reflect_on_all_associations.each do |assoc|
assert_nothing_raised("#{assoc.name} caused an error") do
@m.send(assoc.name, true)
end
end
true
end
|
Listing 2 shows the helper that exercises all of the associations on a
class. The assert_working_associations method
simply walks through all of the associations on the class and sends the
model the name of the association. This catch-all ensures that with a
single line of code per model, I can invoke all relationships on all of
our model tests.
Functional and integration tests
Functional tests exercise the user interface through isolated HTTP
requests. The Rails framework makes it easy to invoke single HTTP GET and
POST commands, forming the backbone of the tests. Integration tests are
the same, but they can invoke many HTTP requests back to back. The
principle and structure of the tests is the same. Listing 3 shows a few
basic functional tests.
Listing 3. A simple functional test
require File.dirname(__FILE__) + '/../test_helper'
require 'causes_controller'
class CausesController; def rescue_action(e) raise e end; end
class CausesControllerTest < Test::Unit::TestCase
fixtures :causes, :members, :quotes, :cause_images, :blogs, :blog_memberships
def setup
@controller = CausesController.new
@request = ActionController::TestRequest.new
@response = ActionController::TestResponse.new
end
def test_index
get :index
assert_response :success
assert_template 'index'
assert_not_nil assigns(:causes)
assert_equal Cause.find_all_ordered.size, assigns(:causes).size
end
def test_should_create_blog
assert Cause.find(2).blog.nil?
get :create_blog, :id => 2
assert Cause.find(2).blog.nil?
login_as :bruce
get :create_blog, :id => 2
assert !Cause.find(2).blog.nil?
assert_equal Cause.find(2).name, Cause.find(2).blog.title
end
|
In Listing 3, you can see that the interactions between the test and the
system are all through HTTP GETs and POSTs. The basic flow of a test is
the following:
- Issue a simple HTTP operation.
- Test the impact of the HTTP operation on the system.
In addition, Listing 3's setup method sets up testing scaffolding to
simulate HTTP calls. The testing scaffolding removes requirements for
networking and infrastructure, isolating the test cases to the application
itself.
Stubs
For ChangingThePresent.org, we added a few test helper methods that made it
easy to do things like log in. You can see the login_as :bruce method call in the fifth line of the
test_should_create_blog method in Listing 3.
That invokes the helper in Listing 4, which stubs out the site login
feature, copying the member's ID directly to the session. If you've used
Rails acts_as_authenticated, you know that a
logged-in user will set the value associated with the :user key within the session.
Listing 4. Stubbing out login
def login_as(member)
@request.session[:user] = member ? members(member).id : nil
end
|
Many developers confuse the ideas behind stubbing and mocking. Stubbing
simply replaces a real-world implementation with a simpler implementation.
In Listing 4, the stub replaced our full login system with a simple
substitute. A stub's job is to simulate the real world. Mocks are
not stubs. A mock object, instead, is like a gauge that measures the way
your application uses an interface. I'll discuss stubs in more detail
later and show you a few examples.
Basic concepts
Now you've seen the essence of the Rails out-of-the-box testing
experience. Before I go much further, I should outline a couple of core
decisions: How much, and how fast? As you build an overall testing
philosophy, you will want to pay careful attention to the trade-offs
surrounding coverage and speed.
Coverage
One of the most critical decisions you'll ever make is how much you test.
If you don't test enough, you'll compromise your code quality and often
even slow down your delivery time. You can test too much, too. Test too
much, and you'll possibly miss deadlines that are important to the
business. To make an informed decision about how much to test, you need to
accurately measure how much you're already testing. One of the most
critical measurements for testing is code coverage.
For ChangingThePresent, we use RCov to determine test coverage. I can run
the traditional rake command and get the
traditional trail of dots as a report. I can also run rake test:coverage to get a more complete report that
looks like the one in Listing 5.
Listing 5. Running rake test:coverage with RCov
807 tests, 2989 assertions, 0 failures, 0 errors
+----------------------------------------------------+-------+-------+--------+
| File | Lines | LOC | COV |
+----------------------------------------------------+-------+-------+--------+
|app/controllers/address_book_controller.rb | 142 | 123 | 84.6% |
|app/controllers/admin_controller.rb | 77 | 65 | 93.8% |
|app/controllers/advisor_admin_controller.rb | 86 | 63 | 88.9% |
|app/controllers/advisors_controller.rb | 52 | 42 | 100.0% |
...
|app/models/stupid_gift.rb | 56 | 45 | 100.0% |
|app/models/stupid_gift_image.rb | 10 | 10 | 100.0% |
|app/models/titled_comment.rb | 2 | 2 | 100.0% |
|app/models/upgrade.rb | 13 | 10 | 100.0% |
|app/models/upgrade_item.rb | 3 | 3 | 100.0% |
|app/models/validation_model.rb | 7 | 7 | 100.0% |
|app/models/volunteer_opportunity.rb | 137 | 129 | 93.0% |
|app/models/work_period.rb | 5 | 4 | 100.0% |
+----------------------------------------------------+-------+-------+--------+
|Total | 12044 | 10044 | 81.8% |
+----------------------------------------------------+-------+-------+--------+
81.8% 167 file(s) 12044 Lines 10044 LOC
|
RCov takes a lot longer to run (just under twice as log for our tests),
so I don't run that command all of the time, but when I do, I can tell
exactly how much test coverage that I have on any given file. Better
still, I can open a coverage file in my browser and see exactly which
lines of code my test cases cover. Figure 1 shows an example of a typical
coverage report.
Figure 1. An actual RCov report
When you have numbers, you can start to make some ballpark decisions
about exactly how much you will test. For ChangingThePresent, we've had
fluctuating test coverage statistics, but we're settling on a number
between 80% and 85%. As new major features are under development, the
coverage will temporarily decrease. As we bring those new features online, our coverage will increase. Presently, the coverage sits at 81.7%.
Keep in mind that our answer may not ultimately be the same as yours.
Test coverage will vary based on the experience of the developers on your
team, the complexity of your application, the tolerance of your
application for errors, and the tolerance of your business for delay. If
you are building an airplane engineering application, you're going to need
more tests; if you're building a fad Web 2.0 application for Facebook that
will be worthless two months from now unless you hit a market window,
you'll need to test less. The best Ruby programmers I know all suggest
coverage of production code of over 80%, and some strive for 100%
coverage.
Even if you do achieve 100% coverage, you have no guarantee that your
tests are any good at all. You have to also consider the types of tests,
with both happy paths and boundary conditions, to have the best possible
coverage.
Now that you have the tools to understand how much to test, you can
switch gears and address testing speed. And with Rails, the database will
determine the speed of your tests. Traditional attempts at database-backed
test tools are riddled with problems. The two biggest problems are
repeatability and speed. From a repeatability standpoint, it's hard to
build a good test suite without changing the database, but changing the
database also changes the test data, which in turn changes the behavior of
your tests. The second problem is speed. Changing the database is
expensive.
Speed
As you know, the Rails environment solves the repeatability problem with
fixtures. Each developer sets up fixtures with test data. Before each test
case, the Rails testing frameworks will completely erase each model's data
and load each fixture that you specify for each test case. Each test case
can then start with a clean slate. But each test case has multiple
individual tests, and each one should be totally independent of the
others. Loading the whole set of fixtures for each individual test would
be way too slow.
Rails partially solves the speed problem with a brilliant compromise.
After running each test case, Rails rolls back all database
changes. A rollback is much faster than loading all fixture data from
scratch. Still, you can't deny the cost of database access. Even with
rollbacks, database-backed testing is slow. And if tests are too slow,
your developers will not run them. If they don't get run, they are
practically useless. Though Rails solves the repeatability issue, it
cannot completely solve the speed issue. The speed of running tests will
shape testing strategies for years to come.
One alternative is to use an in-memory database for your tests. Usually,
SQLite will run much faster than MySQL. The downside is that you will
probably not test on the same platform as your production system.
If you're using ActiveRecord for your database backing, you will likely
do all of your unit tests with database-backed testing. You'll accept the
low speed as a cost of development. But there's no rule that says you have
to use database-backed models to do your functional tests. Many Rails
developers now use stubs or mocks to take the database out of the picture,
forming lightning-fast functional tests.
Mocking and stubbing with Mocha and FlexMock
Earlier, I explained that stubbing is a technique that replaces a
real-world implementation with a simpler implementation. Test cases may
use stubbing to make an implementation simpler, faster, or more
predictable. For example, you might want a system clock to always return
the same time so that your test has repeatable results that you can
verify.
The Mocha framework makes stubbing easy. You just define the result that
you expect. The following code will stub the system class Date to always
return the same date, say Ground Hog Day, as in Listing 6.
Listing 6. Creating a simple stub
ground_hog_day = Date.new(2007, 2, 2)
Date.stubs(:today).returns(ground_hog_day)
assert_equal 2, Date.today.day
|
If a stub provides a simplified simulation of the real world, a mock does
more. Sometimes, simply aping the real world is not enough. When you test,
you want to make sure that your code uses an API correctly. For example,
you might want to verify that a native database application opened a
connection, executed a query, and then closed the connection. You might
want to verify that a controller actually calls save on a model object. So a mock object must
establish expectations, as well as behavior.
Rails has at least three mocking libraries getting at least some play:
Mocha, FlexMock, and RSpec. I'm going to focus on Mocha, but each of the
three has its own set of advantages. Using Mocha, you actually spell out
each expected API call, followed by the result Mocha should return, as in
Listing 7.
Listing 7. The Mocha mocking library
mock_member = mock("member_67")
mock_member.expects(:valid?).returns(true)
mock_member.expects(:save).returns(true)
mock_member.expects(:valid_captcha=).with(true)
mock_member.expects(:plaintext_password).returns('password')
mock_member.expects(:id).returns(67)
Member.expects(:find_by_login).with(nil).returns(mock_member)
post :create, :member => {}, :nonprofit => {:id => 67}
...
assert_response :redirect
assert_redirected_to :controller => 'nonprofits', :action => 'show',
:id => mock_nonprofit_id
|
Listing 7 shows an example test case for creating a new member. I can
establish my expectations for each interaction between the controller and
the mock user. I create a mock member and define each interaction
independently. Next, I actually mock the Member class and mock a finder
that returns mock_member.
You can see that the interaction with the model layer is fairly involved,
but the mock object will completely insulate the member behavior from the
functional test case. This has a couple of immediate advantages. Using
certain APIs, like credit card checkouts and self-destruct switches is not
practical. Other APIs, like time- or memory-based services, are not
predictable enough. You will almost always use mock object frameworks to
mock or stub them.
The more interesting discussion deals with whether to mock or stub your
database-backed model. One upside is speed: This test case will not hit
the database at all. Another is independence. I completely isolate the
code under test to the controller layer. But you can probably notice a few
downsides, too. I do complicate my test cases tremendously. I also force
rippling changes whenever I change the behavior of my models, because I
must change the model object and the test cases that surround them. I can
make it easy for a test case to miss something vital. Adding a single
validation could break an important scenario, but go completely unnoticed.
For this reason, for ChangingThePresent, we do not mock model object
classes. We limit the mocking to external interfaces, such as Web services
to third parties or network services.
I should point out that the Ruby community is heavily leaning in the
direction of the mocking strategies. My shop is definitely cutting against
the grain with our decision to stay database-backed. We've tried both. We
use these techniques because they work better for us and our code base.
Continuous integration
One of the most important enhancements we've added to our overall
discipline is continuous integration (CI). We run the Ruby version of
Cruise Control. Our CI server checks out a clean build and runs test cases
from scratch each time I check in a new change. This server notifies each
developer whenever a change breaks the build. This server allows me to run
a few representative tests before checking in. I might change a few lines
in Member, and then run unit/member_test.rb and
functional/members_controller_test.rb. Fifteen
seconds later, I can check in, confident that Cruise Control will shout
out to me if anything is amiss.
Wrapping up
A few years ago, the interesting debates surrounding testing seemed to
revolve around whether automated testing was a good idea. Now, the debates
are far more interesting:
- Should all of your tests be database-backed, or should you use mocks and stubs to isolate functional tests?
- Is 100% coverage a realistic goal?
- Does the additional speed of an in-memory test offset the additional risk?
My best advice to you is to pick techniques for your team based on what
works for you, and for your customers. Don't let some expert talk you into
something that feels wrong. As always, over the next few years, we will
find that we did not, in fact, have all of the answers. New techniques
will emerge, and existing ones will fall out of favor. What happens on
paper does not always perfectly mirror real-world Rails.
Resources Learn
-
ChangingThePresent.org is
the portal for donors and nonprofits that forms the basis of this article.
You can give a donation gift — an hour of a cancer researcher's time or
making a blind person see — to honor those you love.
-
Learn more about continuous integration
with CruiseControl.rb. The ThoughtWorks continuous integration server
has saved us countless hours of pain.
-
Mocks are not stubs: Martin
Fowler's timeless classic walks you through the difference between stubs
and mock objects.
-
FlexMock is the
Ruby mock framework that seems to be in favor today.
-
Mocha is the mocking framework
that we use for ChangingThePresent.
-
RCov is a great test
coverage tool for Ruby.
-
Get your hands on more howto articles from the Web
development zone's technical library.
-
Subscribe to
the developerWorks Web development newsletter.
Get products and technologies
Discuss
About the author | 
| | Bruce Tate is a father, mountain biker, and kayaker in Austin, Texas. The CTO of WellGood, LLC and the chief architect behind ChangingThePresent.org, he's also the author of nine books, including Beyond Java, From Java to Ruby, and Ruby on Rails: Up and Running. He spent 13 years at IBM and later formed the RapidRed consultancy, where he specialized in lightweight development strategies and architectures based on Ruby, and in the Ruby on Rails framework. He now works with a team of Rails developers to build and maintain the charity portal, ChangingThePresent.org. |
Rate this page
| |
