Category Archives: Automated Testing

What level of test coverage should I shoot for?

“What level of test coverage should I shoot for?” is one of the questions most commonly asked by beginners to Rails testing.

My answer is that you shouldn’t shoot for a particular level of test coverage. I recommend that instead you make testing a habitual part of your development workflow. A healthy level of test coverage will flow from there.

I also want to address why people ask this question. I think people ask this because they want some way of knowing whether they’re testing “enough” or doing testing “right”. Test coverage is one way of measuring this but I think there are better, more meaningful ways.

I think that if you’re feeling the kinds of pains that missing tests leave in their absence, then you need more tests. If you’re not feeling those kinds of pains, then you’re good.

Pains that tell you your test coverage might be insufficient

Too many bugs

This is the obvious one. All software has bugs, but if you feel like the rate of new bugs appearing in production is unacceptably high, it may be a symptom of too little test coverage.

Too much manual testing

This is another fairly obvious one. The only alternative to using automated tests, aside from not testing at all, is to test manually.

Some level of manual testing is completely appropriate. Automated tests can never replace, for example, exploratory testing done by a human. But humans should only carry out the testing that can’t be done better by a computer. Otherwise testing is much more expensive and time-consuming than it needs to be.

Infrequent deployments

Infrequent deployments can arise as a symptom of too few tests for a couple different reasons.

One possible reason is that the need for manual testing bottlenecks the deployment timing. If it takes two days for manual testers to do a full regression test on the application, you can of course only deploy a fully-tested version of your application once every two days at maximum. (And this is assuming the test suite passes every time, which is not typically the case.)

Another possible reason for infrequent deployments is the following logic: things go wrong every time we deploy, therefore things will go wrong less often if we deploy less often, so let’s deploy less often. Unfortunately this decision means that problems pile up and get introduced to production all at once on each deployment instead of getting sprinkled lightly over time.

With the presence of a good test suite, deployments can happen many times a day instead of just once every few weeks or months.

Inability to refactor or make big changes

When a particular change has a small footprint, manual testing is usually good enough (although of course sometimes changes that seem like they’d have small footprints cause surprising regressions in distant areas).

When a change has a large footprint, like a Rails version upgrade or a broad refactoring, it’s basically impossible to gain sufficient confidence of the safety of the change without having a solid automated test suite. So on codebases without good test coverage, these types of improvements tend not to happen.

Poor code quality

As I’ve written elsewhere, it’s not possible to have clean, understandable code without having tests.

The reason is that refactoring is required in order to have good code and automated tests are required in order to do sufficient refactoring.

Diminished ability to hire and retain talent

Lastly, it can be hard to attract and retain high-quality developers if you lack tests and you’re suffering from the ailments that result from having poor test coverage.

If a job candidate asks detailed questions about your development practices or the state of your codebase, he or she might develop a negative perception of your organization relative to the other organizations where he or she is interviewing. All other things being equal, a sophisticated and experienced engineer is probably more likely to pick some other organization that does write tests over yours which doesn’t.

Even if you manage to get good people on your team, you might have trouble keeping them. It’s painful to live with all the consequences of not having tests. Your smartest people are likely to be the most sensitive to these pains, and they may well seek somewhere else to work where the development experience is more pleasant.

The takeaway

I don’t think test coverage is a particularly meaningful way to tell whether you’re testing enough. Instead, assess the degree to which you’re suffering from the above symptoms of not having enough tests. Your degree of suffering is probably proportionate to your need for more tests.

If you do this, then “good” coverage numbers are likely to follow. Last time I checked my main codebase at work my test coverage level was 96.47%.

Which test framework should I learn, RSpec or Minitest?

5 Replies

A common Rails testing question is which testing framework to use. RSpec and Minitest are the two that most people are deciding between. To many beginners it’s not clear which is the better choice.

We could weigh the technical pros and cons of each framework. Many people find things to love and hate about both RSpec and Minitest. You can find some passionate flame wars online if you look.

But before we get into all that, there are some realities to consider that overshadow the relative technical merits of the two frameworks. There are two particular facts we should think about.

Fact #1: usually, someone else decides for you

Most of us don’t have much choice as to whether to use RSpec or Minitest at work.

At some point we’ll get a job. At that job they’ll either use RSpec there or Minitest (or something else or nothing at all). Whatever they use at work, that’s what we’ll be using. Our personal preferences are moot.

Fact #2: usually, they’ve chosen RSpec

For better or worse, it’s my experience and the experience of most Rails developers I’ve talked with that most commercial projects use RSpec. (Note how I said most commerical projects. Most commercial projects use RSpec and most OSS Ruby projects, in my experience, use Minitest. I do not know why this is the way it is.)

Out of curiosity I did a (totally unscientific) poll regarding which test framework they use at work. Take it with a grain of salt, but here are the results.

Even if my numbers are off by quite a bit, RSpec is still the more popular framework.

What does this mean?

My take is that this means if your goal is to get a Rails job, learning RSpec over Minitest will give you a higher probability that your skills match the tech stack that’s used at any particular company.

Some people may object to this way of looking at it. They might argue that if you always you go with whatever’s most popular instead of what’s the best technical choice, you may end up using a Windows laptop or switching from Rails to Node.js.

This argument is flawed though. We’re free to make our own choices on the big things but we can’t dictate what comes along with those choices. We can choose to use Rails instead of a different framework, but we can’t reasonably say that we’re only going to work on Rails projects that use, for example, Minitest and MySQL and Angular and no other combination of technologies. We have to compromise a little or face extremely limited job options.

Also, it doesn’t matter much

Having said all that, I actually don’t believe your choice of which test framework to learn matters!

RSpec and Minitest differ syntactically but they don’t really have meaningful conceptual differences. The principles of testing are the same regardless of which test framework you’re using, or even which language you’re using for that matter.

You’re very unlikely to become an expert in Minitest and then get turned down for a job because they use RSpec there, or vice versa. Employers typically realize that if someone is skilled with testing, they’ll be able to pick up any test framework relatively easily.

So try both

In a sense it might sound depressing that the answer to the RSpec/Minitest question is a) we don’t have a choice and b) it doesn’t matter anyway. I actually find these facts freeing.

If the choice between RSpec and Minitest doesn’t matter that much then we’re free to evaluate both according to our own independent judgment and taste and not worry about whether we’re making the “right” choice. Whatever we choose, we’re likely to develop skills that will apply to any job, whether they use Minitest or RSpec there.

So my advice is to try both frameworks and see which one you like better. Neither one is objectively superior to the other.

But if you just want me to pick for you, I say RSpec

My very simplistic logic is that RSpec is what you’ll most likely be forced to use at work, so that’s what you might as well learn.

But again, I encourage you to try both and decide for yourself. This is ultimately not a very important decision. Learning testing principles is much more important than learning testing frameworks.

How I set up a Rails application for testing

2 Replies

Below is how I set up a fresh Rails application for testing. I’ll describe it in three parts:

An application template that can add all the necessary gems and configuration
My setup process (commands I run to create a new Rails app)
A breakdown of the gems I use

Let’s start with the application template.

My application template

First, if you don’t know, it’s possible to create a file called an application template that you can use to create a Rails application with certain code or configuration included. This is useful if you create a lot of new Rails applications with parts in common.

Here’s an application template I created that will do two things: 1) install a handful of testing-related gems and 2) add a config file that will tell RSpec not to generate certain types of files. A more detailed explanation can be found below the code.

gem_group :development, :test do
  gem 'rspec-rails'
  gem 'factory_bot_rails'
  gem 'capybara'
  gem 'webdrivers'
  gem 'faker'
end

initializer 'generators.rb', <<-CODE
  Rails.application.config.generators do |g|
    g.test_framework :rspec,
      fixtures:         false,
      view_specs:       false,
      helper_specs:     false,
      routing_specs:    false,
      request_specs:    false,
      controller_specs: false
  end
CODE

The first chunk of code will add a certain set of gems to my Gemfile. A more detailed explanation of these gems is below.

The second chunk of code creates a file at config/initializers/generators.rb. The code in the file says “when a scaffold is generated, don’t generate files for fixtures, view specs, helper specs, routing specs, request specs or controller specs”. There are certain kinds of tests I tend not to write and I don’t want to clutter up my codebase with a bunch of empty files. That’s not to say I never write any of these types of tests, just sufficiently rarely that it makes more sense for me to create files manually in those cases than for me to allow files to get generated every single time I generate a scaffold.

The setup process

When I run rails new, I always use the -T flag for “skip test files” because I always use RSpec instead of the Minitest that Rails comes with by default.

Also, incidentally, I always use PostgreSQL. This choice of course has little to do with testing but I’m including it for completeness.

In this particular case I’m also using the -m flag so I can pass in my application template. Application templates can be specified using either a local file path or a URL. In this case I’m using a URL so that you can just copy and paste my full rails new command as-is if you want to.

$ rails new my_project -T -d postgresql \
  -m https://raw.githubusercontent.com/jasonswett/testing_application_template/master/application_template.rb

Once I’ve created my project, I add it to version control. (I could have configured my application template to do this step manually, but I wanted to explicitly show it as a separate step, partially to keep the application template clean and easily understandable.)

$ git add .
$ git commit -a -m'Initial commit'

The gems

Here’s an explanation of each gem I chose to add to my project.

rspec-rails

RSpec is one of the two most popular test frameworks for Rails, the other being Minitest.

The rspec-rails gem is the version of the RSpec gem that’s specifically fitted to Rails.

factory_bot_rails

Factory Bot is a tool for generating test data. Most Rails projects that use RSpec also use Factory Bot.

Like rspec-rails, factory_bot_rails is a Rails-specific version of a more general gem, factory_bot.

capybara

Capybara is a tool for writing acceptance tests, i.e. tests that interact with the browser and simulate clicks and keystrokes.

The underlying tool that allows us to simulate user input in the browser is called Selenium. Capybara allows us to control Selenium using Ruby.

webdrivers

In order for Selenium to work with a browser, Selenium needs drivers. There are drivers for Chrome, drivers for Edge, etc. Unfortunately it can be somewhat tedious to keep the drivers up to date. The webdrivers gem helps with this.

faker

By default, Factory Bot (the tool for generating test data) will give us factories that look something like this:

FactoryBot.define do
  factory :customer do
    first_name { "MyString" }
    last_name { "MyString" }
    email { "MyString" }
  end
end

This is fine for just one record but becomes a problem if we have multiple records plus a unique constraint. If in this example we require each customer to have a unique email address, then we’ll get a database error when we create two customer records because the email address of MyString will be a duplicate.

One possible solution to this problem is to replace the instances of "MyString" with something like SecureRandom.hex. I don’t like this, though, because I often find it helpful if my test values resemble the kinds of values they’re standing in for. With Faker, I can do something like this:

FactoryBot.define do
  factory :customer do
    first_name { Faker::Name.first_name }
    last_name { Faker::Name.last_name }
    email { Faker::Internet.email }
  end
end

This can make test problems easier to troubleshoot than when test values are simply random strings like c1f83cef2d1f74f77b88c9740cfb3c1e.

Honorable mention

I also often end up adding the VCR and WebMock gems when I need to test functionality that makes external network requests. But in general I don’t believe in adding code or libraries speculatively. I only add something once I’m sure I need it. So I typically don’t include VCR or WebMock in a project from the very beginning.

Next steps

After I initialize my Rails app, I usually create a walking skeleton by deploying my application to a production and staging environment and adding one small feature, for example the ability to sign in. Building the sign-in feature will prompt me to write my first tests. By working in this way I front-load all the difficult and mysterious work of the project’s early life so that from that point on, my development work is mostly just incremental.

If you’re brand new to Rails testing and would like to see an example of how I would actually write a test once I have the above application set up, I might recommend my Rails testing “hello world” post.

Why I don’t use Shoulda matchers

7 Replies

One of the testing questions I commonly get is about Shoulda matchers. People ask if I use Shoulda matchers and if Shoulda matchers are a good idea.

I’ll share my thoughts on this. First I’ll explain what Shoulda is, then I’ll explain why I don’t use it.

What Shoulda is

If you’re unfamiliar with Shoulda matchers, the premise, from the GitHub description, is: “Shoulda Matchers provides RSpec- and Minitest-compatible one-liners to test common Rails functionality that, if written by hand, would be much longer, more complex, and error-prone.”

A few examples of specific Shoulda matchers are validates_presence_of (expects that a model attribute has a presence validator), have_many (expects that a has_many association exists), and redirect_to (expects that a redirection takes place).

I like the idea of a library that can clean up a lot of my repetitive test code. Unfortunately, Shoulda matchers only apply to the kinds of tests I would never write.

Test behavior, not implementation

To me it doesn’t make much sense to, for example, write a test that only checks for the presence of an Active Record association and doesn’t do anything else.

If I have an association, presumably that association exists in order to enable some piece of behavior, or else it would be pointless for the association to exist. For example, if a User class has_many :posts, then that association only makes sense if there’s some Post-related behavior.

So there are two possibilities in light of testing that the User class has_many :posts. One is that I write a test for both the association itself and the behavior enabled by the association, in which case the test for the association is redundant and adds no value. The other possibility is that I write a test only for the post association, but not for the post behavior, which wouldn’t make much sense because why wouldn’t I write a test for the post behavior?

To me it only makes sense in this example to write tests for the post behavior and write no tests directly for the association. The logic of this decision can be proved by imagining what would happen if the has_many :posts line were removed. Any tests for the post behavior would start failing because the behavior would be broken without the association line present.

Takeaway

Don’t test low-level implementations. It’s pointless. Test behavior instead.

Since Shoulda is only good for testing low-level implementations, I don’t recommend using it.

Why use Factory Bot instead of creating test data manually?

12 Replies

I recently taught a Rails testing class where we wrote tests using RSpec, Capybara, Factory Bot and Faker.

During the class, one of the students asked, why do we use Factory Bot? What’s the advantage over creating test data manually?

The answer to this is perhaps most easily explained with an example. I’m going to show an example of a test setup that creates data manually, then a test setup that uses Factory Bot, so you can see the difference.

In both examples, the test setup is for a test that needs two Customer records to exist. A Customer object has several attributes and a couple associations.

Non-Factory-Bot example

Here’s what it looks like when I create my two Customer records manually.

The upside to this approach is that there’s no extra tooling involved. It’s just Active Record. There are a few downsides though.

First, it’s wasteful and tedious to have to think of and type out all this fake data (e.g. “555-123-4567”).

Second, it’s going to be unclear to an outside reader what data is significant and what’s just arbitrary. For example, is it significant that John Smith lives in Minneapolis or could his address have been anywhere?

Third, all this test data adds noise to the test and makes it an Obscure Test. When there’s a bunch of test data in the test it makes it much harder to tell at a glance what the essence of the test is and what behavior it’s testing.

This example is tedious enough with only two associations on the Customer model (State and User). You can imagine how bad things might get when you have truly complex associations.

RSpec.describe Customer do
  before do
    @customers = Customer.create!([
      {
        first_name: 'John',
        last_name: 'Smith',
        phone_number: '555-123-4567',
        address_line_1: '123 Fake Street',
        city: 'Minneapolis',
        state: State.create!(name: 'Minnesota', abbreviation: 'MN'),
        zip_code: '55111',
        user: User.create!(
          email: 'john.smith@example.com',
          password: 'gdfkgfgasdf18233'
        )
      },
      {
        first_name: 'Kim',
        last_name: 'Jones',
        phone_number: '555-883-2283',
        address_line_1: '338 Notreal Ave',
        city: 'Chicago',
        state: State.create!(name: 'Illinois', abbreviation: 'IL'),
        zip_code: '60606',
        user: User.create!(
          email: 'kim.jones@example.com',
          password: 'eejkgsfg238231188'
        )
      }
    ])
  end
end

Factory Bot example

Here’s a version of the test setup that uses Factory Bot. It achieves the same result, the creation of two Customer records. The code for this version is obviously much more concise.

RSpec.describe Customer do
  before do
    @customers = FactoryBot.create_list(:customer, 2)
  end
end

This simple code is made possible through factory definitions. In this case there are three factory definitions: one for Customer, one for State and one for User.

In all three of the factory definitions I’m using an additional gem called Faker. Faker helps with the generation of things like random names, phone numbers, email addresses, etc.

Here are the three factory definitions.

FactoryBot.define do
  factory :customer do
    first_name { Faker::Lorem.characters(10) }
    last_name { Faker::Lorem.characters(10) }
    phone_number { Faker::PhoneNumber.cell_phone }
    address_line_1 { Faker::Lorem.characters(10) }
    city { Faker::Lorem.characters(10) }

    # This line will generate an associated State record
    # using the factory definition for State
    state

    # Same here, but for User
    user
  end
end

FactoryBot.define do
  factory :state do
    name { Faker::Lorem.characters(10) }
    abbreviation { Faker::Lorem.characters(10) }
  end
end

FactoryBot.define do
  factory :user do
    email { Faker::Internet.email }
    password { Faker::Internet.password }
  end
end

Takeaways

If you were wondering why exactly we use Factory Bot, the answer is that it makes our tests more convenient to write and more understandable to read. In addition to Factory Bot, the Faker gem can help take away some of the tedium of having to create test data values.

There’s also one other popular method of creating test data which is to use fixtures. Fixtures have the advantage of speeding up a test suite because they’re only loaded once at the beginning of the test suite run (as opposed to factories which are typically run once per test) but I prefer factories because I feel they make tests easier to understand. You can read more about fixtures vs. factories here.

How I set up Factory Bot on a fresh Rails project

7 Replies

A reader of mine recently asked me how I set up Factory Bot for a new Rails project.

There are four steps I go through to set up Factory Bot.

Install the factory_bot_rails gem
Set up one or more factory definitions
Install Faker
Add the Factory Bot syntax methods to my rails_helper.rb file

Following are the details for each step.

Install the factory_bot_rails gem

The first thing I do is to include the factory_bot_rails gem (not the factory_bot gem) in my Gemfile. I include it under the :development, :test group.

Here’s a sample Gemfile from a project with only the default gems plus a few that I added for testing.

Remember that after you add a gem to your Gemfile you’ll need to run bundle install in order to actually install the gem.

source 'https://rubygems.org'
git_source(:github) { |repo| "https://github.com/#{repo}.git" }

ruby '2.7.0'

# Bundle edge Rails instead: gem 'rails', github: 'rails/rails'
gem 'rails', '~> 6.0.2', '>= 6.0.2.2'
# Use postgresql as the database for Active Record
gem 'pg', '>= 0.18', '< 2.0'
# Use Puma as the app server
gem 'puma', '~> 4.1'
# Use SCSS for stylesheets
gem 'sass-rails', '>= 6'
# Transpile app-like JavaScript. Read more: https://github.com/rails/webpacker
gem 'webpacker', '~> 4.0'
# Turbolinks makes navigating your web application faster. Read more: https://github.com/turbolinks/turbolinks
gem 'turbolinks', '~> 5'
# Build JSON APIs with ease. Read more: https://github.com/rails/jbuilder

gem 'devise'

# Reduces boot times through caching; required in config/boot.rb
gem 'bootsnap', '>= 1.4.2', require: false

group :development, :test do
  gem 'pry'
  gem 'rspec-rails'
  gem 'capybara'
  gem 'webdrivers'
  gem 'factory_bot_rails'
end

group :development do
  # Access an interactive console on exception pages or by calling 'console' anywhere in the code.
  gem 'web-console', '>= 3.3.0'
  gem 'listen', '>= 3.0.5', '< 3.2'
  # Spring speeds up development by keeping your application running in the background. Read more: https://github.com/rails/spring
  gem 'spring'
  gem 'spring-watcher-listen', '~> 2.0.0'
end

# Windows does not include zoneinfo files, so bundle the tzinfo-data gem
gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby]

Set up one or more factory definitions

Factory definitions are kind of the “templates” that are used for generating new objects.

For example, I have a user object that needs an email and a password, then I would create a factory definition saying “hey, make me a user with an email and password”. The actual code might look like this:

FactoryBot.define do
  factory :user do
    email { 'test@example.com' }
    password { 'password1' }
  end
end

Factory Bot is smart enough to know that when I say factory :user do, I’m talking about an Active Record class called User.

There’s a problem with this way of defining my User factory though. If I have a unique constraint on the users.email column in the database (for example), then I won’t ever be able to generate more than one User object. The first user’s email address will be test@example.com (no problem so far) but then when I go to create a second user, its email address will also be test@example.com, and if I have a unique constraint on users.email, the creation of this second record will not be allowed.

We need a way of making it so the factories’ values can be unique. One way, which I’ve done before, is to append a random number to the end of the email address, e.g. "test#{SecureRandom.hex}@example.com". There’s a different way to do it, though, that I find nicer. That way is to use another gem called Faker.

Install Faker

Just like I showed with factory_bot_rails above, the Faker gem can be added by putting it into the :development, :test group of the Gemfile.

Then we can change our User factory definition as follows.

FactoryBot.define do
  factory :user do
    email { Faker::Internet.email }
    password { Faker::Internet.password }
  end
end

This will give us random values like eldora@jones.net and lazaromertz@ko.name.

Add the Factory Bot syntax methods to my rails_helper.rb file

The syntax for actually using a Factory Bot factory in a test is as follows:

FactoryBot.create(:user)

There’s nothing wrong with this, but I find that these FactoryBot are so numerous in my test files that their presence feels a little noisy.

There’s a way to make it so that instead we can just write this:

create(:user)

The way to do that is to add a bit of code to spec/rails_helper.rb.

RSpec.configure do |config|
  config.include FactoryBot::Syntax::Methods
end

(You don’t actually add the RSpec.configure do |config| to the spec/rails_helper.rb file. It’s already there. I’m just including it here to show that that’s the block inside of which the config.include FactoryBot::Syntax::Methods line goes.)

What to do next

If you’re curious how to put Factory Bot together with the other testing tools to write some complete Rails tests, I might suggest my Rails testing “hello world” tutorial using RSpec and Capybara.

The difference between system specs and feature specs

1 Reply

If you’re like me, you might have found the difference between RSpec’s “feature specs” and “system specs” to be a little too nuanced to be easily understandable. Here’s my explanation of the difference between the two.

Two levels of Rails tests

For some background, I want to talk about the main two types of Rails tests that I use, and where feature specs, the predecessor to system specs, come intro the picture.

I think of Rails tests as existing on two basic levels.

High-level, course-grained

One level of tests is at a high level. These tests simulate a user visiting pages, filling in forms, clicking links and buttons, etc. Different people use different terms for these tests including “integration test”, “end-to-end test” and “acceptance test”.

Because these types of tests are often expensive to write and run, they tend not to cover every nook and cranny of the application’s behavior.

In RSpec terminology, these types of tests have historically been called feature specs.

Low-level, fine-grained

The other level of tests is at a lower level. There are all kinds of tests you could possibly write with Rails/RSpec (model specs, request specs, view specs, helper specs) but I tend to skip most of those in most scenarios and only write model specs.

From feature spec to system spec

The backstory

When Rails 5.1 came out in April 2017, one of the features it introduced was system tests. Here’s why this was done.

By default Rails ships with Minitest. The inclusion of Minitest provides a way to write model tests and certain other kinds of tests, but it historically hasn’t provided a way to write full-blown end-to-end without doing some extra work yourself, like bringing Capybara and Database Cleaner into the picture. This is the rationale for Rails 5.1’s addition of system tests, in my understanding. (To be clear, we’re still talking about system tests, not system specs.)

System specs wrap system tests

According to the RSpec docs, “System specs are RSpec’s wrapper around Rails’ own system tests.” This means that it’s no longer required to explicitly include the Capybara gem, and because system tests are already run inside a transaction, you don’t need Database Cleaner.

Summary

System specs are a wrapper around Rails’ system tests. The benefits of using system specs instead of feature specs is that you don’t have to explicitly include the Capybara gem, nor do you have to use Database Cleaner.

What kinds of Rails tests I write and what kinds I don’t

3 Replies

The challenge of deciding what kinds of tests to write

There are a lot of different kinds of tests a developer could possibly write. In RSpec there are model specs, feature specs, view specs, helper specs, routing specs, controller specs, and request specs. Do you need to write all these different types of tests? If not, which ones should you skip?

The two types of tests I write

In my Rails apps I usually only write two kinds of tests. To use framework-agnostic language, I write model tests and acceptance tests. To use RSpec-specific language, I write model specs and feature specs. Let’s talk about each of these two in a little more detail.

Model tests are relatively isolated. A model test will test the behavior of a particular Ruby class in my Rails application, e.g. the Customer model. Model tests are what give me confidence that my features are working at a fine grain. (I often use the terms “class” and “model” somewhat interchangeably since a Rails model takes the form of a Ruby class.) For example, if I have a method that returns the grand total of the payments a certain customer has made, I’ll write a model test for that method.

What about acceptance tests? Whereas model tests are relatively isolated, acceptance tests are the opposite; they’re relatively integrated. An acceptance test exercises the whole stack, browser and all. Acceptance tests give me confidence that everything works together. For example, if I have CRUD functionality for a Customer resource, I’ll write acceptance tests for creating a customer, updating a customer, and deleting a customer.

A note on terminology

As of the time of this writing there not complete consensus in the testing world on testing terminology (and I don’t expect this to change anytime soon). For example, some developers might consider the terms end-to-end-test, integration test, and acceptance test to refer to basically the same thing, whereas other developers might consider these three terms to refer to three completely distinct types of tests.

What does this mean for you in terms of your testing journey? It means that if you, for example, see a reference to end-to-end tests that doesn’t square with your understanding of the term end-to-end tests, that doesn’t necessarily mean that your understanding is mistaken, it just means that not everyone in the testing world talks about the same concepts using the same exact terms. I think understanding this communication challenge is helpful since otherwise one might get distracted by all the apparent discrepancies.

Testing terminology + RSpec

How do model tests and acceptance tests map to RSpec? Model tests—or model “specs” in RSpec terminology—are the tests that live in the spec/models directory. Nothing but bare RSpec is needed in order to write model specs. What the outside world calls acceptance tests (or again, integration tests or end-to-end tests) are known in RSpec as feature specs. Since feature specs exercise features via the browser, the Capybara library (a library that allows us to manipulate the browser using Ruby) is necessary.

Why, by the way, is an RSpec test called a “spec” instead of a “test”? The idea is that each RSpec file is an “executable specification”. Each test case is actually called an example in RSpec terminology, an example of how that feature is supposed to behave. A set of examples makes up a spec. Personally, I’m willing to play along with this terminology to some extent, but I frankly find it kind of annoying. You’ll find that most of the time I say “test” instead of “spec” or “example”.

The types of tests I don’t write

I typically don’t write view specs, routing specs, or request specs/controller specs. Let’s discuss each of these.

Request specs/controller specs

First, a bit of terminology: request specs and controller specs are two very slightly different things, although for our purposes we can treat the two terms as meaning the same thing. A close enough approximation to the truth is that “controller spec” is the old name and “request spec” is the new name.

I usually don’t write controller/request specs (although I do sometimes). The reason is that I try to have as little code in my controllers as possible. For the most part, my controllers don’t do anything, they just call model objects that do things. Any code that might have been covered by a controller/request spec, I prefer to cover via a feature spec.

View specs, routing specs

I don’t write view specs and I don’t believe I’ve ever encountered anyone who does. The reason is that I’ve never been able to conceive of how a view spec could have value that a feature spec doesn’t already provide.

I also don’t write routing specs, and for the same reason. I find them to be redundant to feature specs.

Conclusion

If you’re just getting started with Rails/RSpec testing and you’re wondering which types of tests to focus on, I would recommend focusing on model specs and feature specs.

Where to start with introducing TDD to a new Rails app

How do I introduce TDD into a new Rails app?

This depends on your experience level with Rails and with testing.

I personally happen to be very experienced with both Rails and testing. When I build a new Rails application, I always start writing tests from the very beginning of the project.

If you’re new to Rails AND new to testing, I wouldn’t recommend trying to learn both at the same time. That’s too much to learn at once. Instead, get comfortable with Rails first and then start learning testing.

What if you’re comfortable with Rails but not testing yet? How do you introduce testing to a new Rails app then?

I would suggest starting with testing from the very beginning. Adding tests to your application is never going to be easier than it will be at the very beginning. In fact, retroactively adding tests to an existing application is notoriously super hard.

The kinds of tests to start with

What kinds of tests should you add at the beginning? I would recommend starting with acceptance tests, also known to some as integration tests, end-to-end tests or, in RSpec terminology, feature specs. The reason I recommend starting with feature specs is that they’re conceptually easy to grasp. They’re a simulation of a human opening a browser and clicking on things and typing stuff.

You can start with a feature spec “hello world” where you just visit a static page and verify that it says “hello world”. Once you’re comfortable with that, you can add feature specs to all your CRUD operations. I have a step-by-step formula for writing feature specs that you can use too.

Once you’re somewhat comfortable with feature specs, you can move on to model specs.

Testing vs. TDD

I’ve been sloppy with my language so far. I’ve been talking about testing but the question was specifically about TDD. How do you introduce TDD into a new Rails app?

First let me say that I personally don’t do TDD 100% of the time. I maybe do it 60% of the time.

I mainly write two types of tests in Rails: feature specs and model specs. When I’m building a new feature, I often don’t know exactly what form the UI is going to take or what labels or IDs all the page elements are going to have. This makes it really hard for me to try to write a feature spec before I write any code. So, usually, I don’t bother.

Another reason I don’t always do TDD is that I often use scaffolds, and scaffolds and TDD are incompatible. With TDD you write the tests first and the code after. Since scaffolds give you all the code up front, it’s of course impossible to write the tests before the code because the code already exists.

So again, the only case when I do TDD is when I’m writing model code, and I would guess this is maybe 60% of the time.

How, then, should you start introducing TDD to a new Rails app? I personally start TDD’ing when I start writing my first model code. The model code is often pretty trivial for the first portion of a Rails app’s lifecycle, so it’s usually a while before I get into “serious TDD”.

What should I be testing vs. what should I assume works because it was tested in some other way? For example, why test generated code?

I’ll answer the second part of this question first.

Why test generated code?

I assume when we say “generated code” we’re mainly talking about scaffolding.

If you’re never ever going to modify the generated code there’s little value in writing tests for it. Scaffold-generated CRUD features pretty much always just work.

However, it’s virtually never the case that a scaffolded feature comes out exactly the way we want with no modification required. For anything but the most trivial models, a little tweaking of the scaffold-generated code is necessary in order to get what we need, and each tweak carries a risk of introducing a bug. So it’s not really generated code. Once a human starts messing with it, all bets are off.

That’s not even the main value of writing tests for generated code though. The main value of the tests covering generated code (which, again, is rarely 100% actual generated code) is that the tests help protect against regressions that may be introduced later. If you have tests, you can refactor freely and be reasonably confident that you’re not breaking anything.

Having said that, I don’t try to test all parts of the code generated by a scaffold. There are some things that are pointless to test.

What should I test and what should I assume works?

When I generate a scaffold, there are certain types of tests I put on the scaffold-generated code. I have a step-by-step process I use to write specific test scenarios.

I typically write three feature specs: a feature spec for creating a record using valid inputs, a feature spec for trying to create a record using invalid inputs, and a feature spec for updating a record. That’s enough to give me confidence that things aren’t horribly broken.

At the model level, I’ll typically add tests for validations (I am TDD’ing in this case) and then add the validations. Usually I only need a few presence validations and maybe a uniqueness validation. It’s not common that I’ll need anything fancier than that at that time.

There are certain other types of tests I’ve seen that I think are pointless. These include testing the presence of associations, testing that a model responds to certain methods, testing for the presence of callbacks, and testing for database columns and indexes. There’s no value in testing these things directly. These tests are tautological. What’s better is to write tests for the behaviors that these things (i.e. these associations, methods, etc.) enable.

Suggestions for further reading

How I approach test coverage metrics

The Beginner's Guideto Rails Testing