Category Archives: Automated Testing

Factories and fixtures in Rails

4 Replies

One necessity of automated testing is having some data to test with. There are three ways I know of to generate test data in Rails:

  • Manually
  • Fixtures
  • Factories

All three can be viable solutions depending on the situation. Let’s first explore manual creation.

Manual data creation

Manual data creation can be convenient enough if you only have a few attributes on a model and no dependencies. For example, let’s say I have a PaymentType model with just one attribute, name. If I want to test both valid and invalid states, that’s easy:

valid_payment_type = PaymentType.new(name: 'Visa')
invalid_payment_type = PaymentType.new(name: '')

But now let’s say we have the idea of an Order which is made up of multiple LineItems and Payments, each of which has a PaymentType.

order = Order.create!(
  line_items: [
    LineItem.create!(name: 'Electric dog polisher', price_cents: 40000)
  ],
  payments: [
    Payment.create!(
      amount_cents: 40000,
      payment_method: PaymentMethod.create!(name: 'Visa')
    )
  ]
)

That’s annoying. We had to expend mental energy to arbitrarily come up with details (“Electric dog polisher” with a price of $400.00, paid by Visa) that aren’t even necessarily relevant.

What if all we wanted to test was that when the payment total equals the line item total, order.balance_due returns zero?

This is where factories come in handy.

Factories

Factories are actually not a test-specific concept. “Factory method” is a design pattern that you can find in the “Gang of Four” design patterns book. The pattern just happens to come in handy for the purpose of testing.

The idea with a factory is basically that you have a method/function that generates new class instances for you.

In the Gang of Four book they use an example where a factory called Creator will return either an instance of MyProduct or YourProduct depending on certain conditions. So rather than having to say “if this case, then instantiate a MyProduct, otherwise, instantiate a YourProduct” all over the place, you can just say Creator.create(relevant_data) and get back the appropriate class instance.

You might be able to imagine how such a factory would be useful. In the case of testing, the kind of factories we want will be a little bit different. We’re not concerned with abstracting away whether class A or class B gets instantiated. We want to abstract away the generation of irrelevant details and tedious-to-set-up model associations.

Here’s an example of how the setup for an Order instance might look if we used a factory, specifically Factory Bot. (By the way, Factory Bot is the most popular Rails testing factory but there’s nothing particularly special about the way it works. The ideas in Factory Bot could have been implemented any number of ways.)

order = FactoryBot.create(
  :order,
  line_items: [FactoryBot.create(:line_item, price_cents: 40000)],
  payments: [FactoryBot.create(:payment, amount_cents: 40000)]
)

In this case we’re specifying only the details that are relevant to the test. We don’t care about the line item name or the payment method. As long as we have a payment total that matches the line item total, that’s all where care about.

How would we achieve this same thing using fixtures?

Fixtures

First off let me say that I’m much less experienced with fixtures than I am with factories. If you want to learn more about fixtures I suggest you read something from someone who knows what they’re talking about.

Having said that, if we wanted to set up the same test data that we did above using factories, here’s how I believe we’d do it with a fixture. Typically fixtures are expressed in terms of YAML files.

# orders.yml
payments_equal_line_item_total:
  # no attributes needed

# line_items.yml
electric_dog_polisher:
  order: payments_equal_line_item_total
  name: 'Electric dog polisher'
  price_cents: 40000

# payment_methods.yml
visa:
  name: 'Visa'

# payments.yml
first:
  order: payments_equal_line_item_total
  payment_method: visa
  amount_cents: 40000

Once the fixture data is established, instantiating an object that uses the data is as simple as referring to the key for that piece of data:

order = orders(:payments_equal_line_item_total)

So, what’s best? Factories, fixtures, or manual creation?

Which is best?

I think I’ve demonstrated that manually generating test data can quickly become prohibitively tedious. (You can often mitigate this problem, though, by using loose coupling and dependency injection in your project.) But just because manual data generation doesn’t work out great in all scenarios doesn’t mean it’s never useful. I still use manual data generation when I only need to spin up something small. It has a benefit of clarity and low overhead.

Like I said above, I’ve definitely used factories more than fixtures. My main reason is that when I use factories, the place where I specify the test data and the place where I use the test data are close to each other. With fixtures, on the other hand, the setup is hidden. If I want to see exactly what test data is being generated, I have to go into the YAML files and check. I find this too tedious.

Another issue I have with fixtures in practice is that, in my experience, teams will set up a “world of data” and then use that big and complicated world as the basis for all the tests. I prefer to start each test with a clean slate, to the extent possible, and generate for each test the bare minimum data that’s needed for that test. I find that this makes the tests easier to understand. I want to be clear that I think that’s a usage problem, though, not an inherent flaw in the concept of fixtures.

There’s also no reason why you couldn’t use both factories and fixtures in a project. I’m not sure what the use case might be but I could imagine a case where fixtures provide some minimal baseline of fixed data (the payment types of Visa, MasterCard, cash, etc. come to mind as something that would be often-needed and never changed) and then factories take the baton from there and generate dynamic data on a per-test basis.

My go-to test generation method is factories, and that’s what I generally recommend to others. But if the use case were right I wouldn’t hesitate to use fixtures instead of or in addition to factories.

What a “walking skeleton” is and why I always start projects with one

Leave a reply

Every software project involves some amount of mystery and uncertainty. In fact, it’s often the case that a software project is hardly made of anything but mystery and uncertainty.

Some jobs are more mysterious than others. I was recently asked by a client to build a Rails application that contained CRUD interfaces for a Patient resource and an InsuranceType resource. That work was not very mysterious.

My client also asked me to set up staging and production environments for this application on AWS. This might sound easy but setting up a Rails application on AWS/Elastic Beanstalk is totally not straightforward, even though it’s 2018, and even though I myself wrote guides on how to get Rails working on AWS. The work of getting the application deployed to AWS was in fact mysterious.

Enter the Walking Skeleton

A Walking Skeleton is basically the thinnest possible slice of all the layers in an application. The idea is that you front-load the difficult/mysterious work in a project so you get it out of the way early while the stress level is low and the remaining budget is high.

I learned about the idea of a Walking Skeleton from Growing Object-Oriented Software, Guided by Tests. The name refers to the idea that your early application hardly does anything but it can still stand up.

The authors’ suggestion as I understood it was to set up both an application’s production environment and testing infrastructure as part of a Walking Skeleton, both of which can be mysterious/time-consuming/difficult to set up.

So in the case of my application, I might get my Elastic Beanstalk environment set up, create just one of my CRUD interfaces (whatever the simplest one is) and write a couple feature specs to exercise my CRUD interface. This will force me to figure out how to get AWS/EB set up (again, non-trivial) as well as install RSpec, Factory Bot, Capybara, etc. From that point on everything is just incremental.

The alternative to a Walking Skeleton

Failure to create a Walking Skeleton can contribute to ghastly project failure. If you’d like to hear one of my horror stories about this you can check out another article of mine, Deploy to Production on Day One.

How dependency injection can make Rails tests easier

Leave a reply

“Dependency injection” is a fancy-sounding term. When I first heard it I assumed it referred to some super-advanced technique. It wasn’t until years later that I realized that dependency injection is a pretty straightforward technique once you understand what it is.

My aim with this post is to cut through the jargon and show you in simple terms what dependency injection is and why it’s useful.

But first: why are we interested in this topic?

Why bother learning dependency injection?

Depending on how it’s written, some code can be easy to test and some code can be hard to test. Code with entangled dependencies is hard to test.

Why is code with entangled dependencies hard to test? Imagine I have a class `Order` that requires instances of a class called `Payment` in order to function. Let’s then imagine that `Payment` needs some `PaymentType` instances (Visa, MasterCard, cash, etc.) in order to work.

This means that in order to test the class I’m interested in, `Order`, I have to bring two other classes into the picture, `Payment` and `PaymentType`, just to perform the test. And what if `Payment` and `PaymentType` in turn depend on other classes? This test is going to potentially be very tedious to set up.

The opposite of having entangled dependencies is having loose coupling and modularity. Modular, loosely coupled code is easy to test. A number of factors have a bearing on how modular and loosely coupled your code will end up. What I want to show you right now is how dependency injection can help make your code more modular and therefore more easily testable.

An dependency-laden Rails model

Let’s say you’re working on a legacy project that you recently inherited. There’s very little test coverage. You encounter an ActiveRecord model called `CustomerFile`. There’s a method called `parse` that evidently parses a CSV.

class CustomerFile < ActiveRecord::Base
  belongs_to :customer

  def parse
    rows = []

    content = File.read(customer.csv_filename)
    CSV.parse(content, headers: true) do |data|
      rows << data.to_h
    end

    rows
  end
end

Let’s focus on this line for a second: `content = File.read(customer.csv_filename)`.

Apparently a `CustomerFile` object has an associated `customer` object which in turn has a `csv_filename`. How exactly does `customer` get set? It’s not clear. Where exactly is the file that `customer.csv_filename` points to? That’s not obvious either.

We can try to write a test for `CustomerFile` but it probably won’t go very well.

RSpec.describe CustomerFile do
  describe '#parse' do
    let(:customer_file) { CustomerFile.new }

    it 'parses a CSV' do
      # How do we know what to expect?
      # Where is the file that "customer.csv_filename" refers to?
      # expected_first_row = ?

      expect(customer_file.parse[0]).to eq(expected_first_row)
    end
  end
end

Our attempt to write a test hasn’t proven very fruitful. The challenge of writing a test for this class is somewhat “uncomeatable”.

The reason it’s hard to write this test is that `CustomerFile` has a dependency inside of a dependency. We don’t know how to make a `customer`, and even more problematic, we don’t know how to make a CSV file for that customer.

Applying dependency injection for easier testability

Let’s imagine now that `parse` doesn’t require that we have a `customer` with `csv_filename` that points to some mysterious file on the filesystem somewhere.

Let’s imagine a version of `parse` that just takes the file contents as an argument.

class CustomerFile < ActiveRecord::Base
  belongs_to :customer

  def parse(content)
    rows = []

    CSV.parse(content, headers: true) do |data|
      rows << data.to_h
    end

    rows
  end
end

When we try to write a test now, we’ll see that it’s much easier.

RSpec.describe CustomerFile do
  describe '#parse' do
    let(:customer_file) { CustomerFile.new }
    let(:content) { "First Name,Last Name\nJohn,Smith" }

    it 'parses a CSV' do
      expected_first_row = {
        'First Name' => 'John',
        'Last Name' => 'Smith'
      }

      expect(customer_file.parse(content)[0]).to eq(expected_first_row)
    end
  end
end

In this case `parse` doesn’t know or care where the CSV content comes from. This means that we don’t have to bring the filesystem into the picture at all which makes writing this test very convenient. No `customer` object or `customer.csv_filename` value necessary.

If we want to use `parse` for real in the application we can just pass in the file contents like this: `parse(File.read(customer.csv_filename))`.

Conclusion

Modular, loosely coupled code is testable code. You can use dependency injection to help make your code more modular.

Atomic commits and testing

Leave a reply

What atomic commits are and why they’re advantageous

I like to make Git commits very frequently. Looking at the Git log for a recent project, it looks like I tend to commit about every 5-15 minutes.

I find that the smaller my commits are the easier I make life for myself. I remember painful occasions in the past where I would do a big chunk of work, maybe two hours or so. Things would be going fine, going fine, and then all the sudden my work would collapse on itself and I wouldn’t be able to figure out how to get it back to a working state.

At that point my options would be either to scratch my head for the next hour to try to figure out what went wrong or to revert the whole two hours’ worth of work. I had painted myself into a corner. When I teach programming classes I see students paint themselves into a corner like this all the time.

This kind of stuff doesn’t really happen to me anymore. One of the main reasons is that I practice atomic commits.

“Atomic commit” is basically a fancy way of saying a commit that commits one and only one thing. It’s a single complete unit of work.

Students of mine often find it funny that I’ll make a commit after changing just one or two lines of code. But a commit is not just something I do after completing a piece of work. A commit is something I do before I start a piece of work.

Let’s say I have Chunk A and Chunk B, two unrelated pieces of work. Chunk A is a 30-second change. Chunk B is a 20-minute change. A lot of people might not bother committing Chunk A because it’s so small. But then Chunk A “comes along for the ride” when I’m working on Chunk B. If I screw up Chunk B and have to bail and revert and start over, then I also end up reverting Chunk A. Or if a week later I find out Chunk B introduced a bug and I need to revert Chunk B at that point, Chunk A gets reverted as well even though it has nothing to do with Chunk B.

These are the things I have in mind when I commit what might seem to others like a comically tiny change.

Atomic commits also make it easier to track down the source of mysterious regressions. Let’s say a feature was known to be working on January 5th and broken on June 11th and nobody knows when exactly the feature broke. Git bisect can make it very quick and easy to find out exactly which commit introduced the regression. At least, it’s easy if the team has been practicing atomic commits. If each commit is huge and contains multiple pieces of work, git bisect loses a lot of its usefulness.

By the way, I think “WIP” (work in progress) is one of the worst possible commit messages. First, it basically screams “This commit is not a complete unit of work.” Second, it’s about as vague as it gets. Committing “WIP” is basically saying “Rather than take 30 seconds to think of a meaningful commit message, I’d rather make you take several minutes to try to figure out what this commit is all about.” Please don’t commit “WIP”.

What atomic commits have to do with testing

I find it advantageous to make sure each commit leaves the application’s test suite in a passing state.

This way you’re free to check out any commit in the application’s history and you can have a reasonable expectation that the test suite will pass. (This becomes less true the further back in history you go, of course, but it’s at least true for the recent commit history which is usually the history you’re most interested in.)

Some developers don’t find it important to keep the test suite passing on every commit. When this is the case I might check out a certain revision and see that the tests don’t pass. Then I’m forced to wonder: is it supposed to be like this or was this an accident? There’s nothing more frustrating than getting tripped up by a failing test, finally asking someone about it, and getting the response, “Oh, yeah, that test is supposed to fail.” Allowing failing tests to be committed to the repo arguably defeats the purpose of having tests.

It’s okay for a commit not to add a test. It’s okay for a commit to add a passing test. But if a commit commits a failing test then the commit is not a complete unit of work, and the benefits of a repo full of atomic commits are to some extent lost.

I also often find it handy to use git bisect in conjunction with an application’s test suite. Using git bisect is all about going back to a certain point in time and asking “Does feature X work at this commit?” Sometimes the test that answers that question is a manual check. Sometimes it’s an automated test. If the team has a habit of making only small, atomic commits, using git bisect together with tests is a lot easier.

Things you can ignore when getting started with Rails testing

2 Replies

Here’s an incomplete list of tools and concepts you might encounter when first trying to learn Rails testing: Capybara, Cucumber, Database Cleaner, factory_bot, Faker, MiniTest, RSpec, system tests, Test::Unit, acceptance tests, end-to-end-tests, mocks, stubs, unit tests and TDD.

That’s a lot of stuff. If you’re like most humans you might look at this long list of things, feel confused about where to start, and say, “I don’t know what to do. I’ll just deal with this later.”

The challenge of getting started with testing would be much easier if you knew exactly what you needed to know and what you could safely ignore. What follows is a list of what you can safely ignore.

Framework Decisions

You don’t need to get hung up on which framework to use. You literally can’t go wrong. The principles of testing are the same no matter which testing framework you use. Plus you can always change your mind later.

When I first got started with Rails, the way I decided on a testing framework was very simple. I noticed that most Rails developers used RSpec, so I just picked RSpec.

(I actually used Test::Unit for a while before realizing most Rails developers used RSpec. So I just switched. It wasn’t a very big deal.)

Cucumber, Capybara and System Tests

Most Rails test suites have two main sub-suites: a suite of model tests and a suite of integration tests.

Model tests test the behavior of ActiveRecord models by themselves. Integration tests actually spin up a browser and do things like fill out forms and click links and buttons as if a human is actually using the application.

Between these two, the bar is lower for model tests. For integration tests you have to do almost all the same stuff as model tests plus more. For this reason I suggest that if you’re a newcomer to Rails testing that you start with model tests (or even just Ruby tests without Rails) and ignore integration tests altogether until you get more comfortable with testing.

So you can ignore Cucumber (which I don’t recommend using at all), Capybara and system tests, which are all integration testing tools.

View Specs

I’ve never written a view spec. To me they seem tautological. I’ve also never encountered any other Rails developer who advocates writing view specs.

If for some reason I had a wildly complicated view, I could see a view spec potentially making sense. But I haven’t yet encountered that case.

Helper Specs

I don’t tend to write helper specs because I don’t tend to write helpers. Helpers are a fairly peripheral area of Rails that you can safely disregard altogether when you’re getting started with Rails testing.

Routing Specs

Like view specs, I find routing specs to be tautological. I don’t write them.

Request Specs

Request specs are a great boon and even necessity if your application has an API or if your controllers do anything non-trivial. But I don’t think you should worry about request specs when you’re just getting started.

Controller Specs

Controller specs are deprecated in favor of request specs.

What Not to Ignore

The main thing I would recommend learning when you’re getting started with Rails testing is model tests.

When you’re learning about model tests you’ll naturally have to get acquainted with RSpec syntax (or whichever framework you choose), factory_bot and Database Cleaner (or analogous tools). But other than the actual testing techniques, that’s about it as far as model test tooling goes.

If you want to make life even easier on yourself you can learn just Ruby and RSpec with no Rails involved. Then, after you get comfortable with RSpec syntax and basic testing techniques, you can approach Rails testing with more confidence.

Why I recommend against using Cucumber

18 Replies

Around the time I first started using Rails in 2011, I noticed that a lot of developers, seemingly all Rails developers, were using Cucumber to assist with testing.

I bought into the idea—describing test cases in plain English—but in practice I found Cucumber not to be very valuable. In fact, my experience has been that Cucumber adds a negative amount of value.

In recent years I’ve noticed (although this is anecdotal and might just be my perception) that fewer codebases seem to use Cucumber and that fewer Rails developers seem to be on board with Cucumber. I had thought Cucumber was pretty much dead. But lately, to my surpise, I’ve seen Cucumber recommended to testing noobs more than a few times. Since I consider Cucumber to be a bad thing, I want to explain why I think so and why I don’t think other people should use it.

In my view there are two general ways Cucumber can be used: it can be used as intended or it can be abused. In the former case, I believe Cucumber has a small negative value. In the latter case I believe it has a large negative value.

Why Cucumber is bad when it’s not used as intended

Most production Cucumber scenarios I’ve seen look something like this:

Given a user exists with email "test@example.com" and password "mypassword"
And I visit "/sign_in"
And I fill in the "Email" field with "test@example.com"
And I fill in the "Password" field with "mypassword"
And I click "Sign In"
And I visit "/user/edit"
And I fill in the "First Name" field with "John"
And I fill in the "Last Name" field with "Smith"
And I fill in the "Age" field with "30"
And I click "Save"
And I visit "/profile"
Then I should see "John Smith, 30"

These kinds of tests, with fine-grained steps, arise when the developers seem to mistake Cucumber for a way to write Ruby in English. The above scenario provides exactly zero benefit, in my opinion, over the following equivalent Capybara scenario:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The Cucumber/Gherkin version is no shorter nor more easily understandable.

To be fair to Cucumber, nobody who understands Cucumber advocates writing Cucumber scenarios in this way. The Cucumber creator himself, Aslak Hellesøy, wrote a post in 2011 saying not to do this. Other people have written similar things.

I think it’s telling that so many people have written blog posts advising against the very common practice of writing fine-grained Cucumber steps. To me it’s kind one of those gas station doors that looks for all the world like a pull door, so every single person who comes up to it pulls it instead of pushes it, feels like a dumbass, and then pushes it. So the gas station manager puts up a big sign that says “PUSH”, but most people don’t notice it and the problem persists. What instead should have been done is to make the push door look like a push door, without the big useless handle that you’re not supposed to pull. I get that the Cucumber maintainers tried to do that by removing `web_steps.rb`, but in my experience it didn’t seem to work.

And it doesn’t matter much anyway because Cucumber still sucks even if you don’t abuse it by writing fine-grained steps. I’ll explain why I think so.

Why Cucumber is bad even when it is used as intended

Here’s a version of the above Cucumber scenario that’s done in the way the Cucumber creators would intend. There are two parts.

First, the Gherkin steps:

Given I am signed in
And I provide my name and age details
Then I should see those details on my profile page

Second, the underlying Ruby steps:

Given /^I am signed in$/ do
  visit sign_in_path
  fill_in 'Email', with: 'test@example.com'
  fill_in 'Password', with: 'mypassword'
  click_on 'Sign In'
end

And /^I provide my name and age details$/ do
  visit edit_user_path
  fill_in 'First Name', with: 'John'
  fill_in 'Last Name', with: 'Smith'
  fill_in 'Age', with: '30'
  click_on 'Save'
end

Then /^I should see those details on my profile page$/ do
  visit profile_path
  expect(page).to have_content('John Smith, 30')
end

This is actually pretty decent-looking and appealing, at least at first glance. There are two problems, though. First, this way of doing things doesn’t really provide any clarity over doing it the Capybara way. Second, the step definitions usually end up in a single, flat file full of “step soup” where unrelated steps are mixed together willy-nilly.

Compare this again with the Capybara version:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The sign in portion is usually abstracted away in Capybara, too, so the scenario would look more like this:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

sign_in

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

That’s not too crazy at all. In order for Cucumber to be a superior solution to using bare Capybara, it would have to have some pretty strong benefits to compensate for the maintenance burden and cognitive overhead it adds. But it doesn’t.

So what do I recommend doing instead of using Cucumber? I think just using Capybara by itself is fine, and better than using Capybara + Cucumber. I also think Capybara + page objects is a pretty good way to go.

Rails testing resource roundup

Leave a reply

.resource-item {
border-bottom: 1px solid #DDD;
padding-bottom: 50px;
margin-top: 30px;
margin-bottom: 50px;
}

Below is a list of testing resources I’ve either used myself or heard recommended. I tried to make the list as Ruby-centric as possible, although some resources from other ecosystems are so good that I didn’t want to exclude them.

I intend this to be a living list that grows over time. If you know of something that should be on this list but isn’t, please let me know, either in a comment on this page or via email.

Disclosure statement: None of the links below are affiliate links, although I do have a relationship with some of the authors/creators of these resources.

Section One: Ruby/Rails Specific Resources

Print Book: Rails 5 Test Prescriptions


Excerpt from Pragmatic Bookshelf summary:

“Does your Rails code suffer from bloat, brittleness, or inaccuracy? Cure these problems with the regular application of test-driven development. You’ll use Rails 5.2, Minitest 5, and RSpec 3.7, as well as popular testing libraries such as factory_bot and Cucumber.”

Details at The Pragmatic Bookshelf

Side note: you can also listen to my interview with author Noel Rappin on the Ruby Testing Podcast.

eBook: Everyday Rails Testing with RSpec


Summary from Leanpub:

“Real-world advice for adding reliable tests to your Rails apps with RSpec, complete with expanded, exclusive content and a full sample application. Updates for 2017 now available—RSpec 3.6, Rails 5.1, and more! Learn to test with confidence!”

Details at Leanpub

Screencast Series: Destroy All Software

Not everything on Destroy All Software (DAS) is testing-related but a lot of it is. I often see DAS recommended when people ask for testing-related resources.

Destroy All Software Catalog

Online Course Series: Upcase’s “Learn Testing” Courses

Summary from Upcase website:

“Test-driven development, or TDD, is the practice of writing your tests firsts, then using those tests to guide you as you write your actual production code. This may sound crazy, but it turns out that it makes writing code much easier. It provides a clear workflow and next steps while you’re building and has the added benefit of producing a test suite you can have confidence in. With these courses and videos we’ll teach you everything you need to know to get started with TDD.”

Details at Upcase

Print Book/eBook: Effective Testing with RSpec 3

Excerpt from Pragmatic Bookshelf summary:

“This definitive guide from RSpec’s lead developer shows you how to use RSpec to drive more maintainable designs, specify and document expected behavior, and prevent regressions during refactoring. Build a project using RSpec to design, describe, and test the behavior of your code. Whether you’re new to automated tests or have been using them for years, this book will help you write more effective tests.”

Details at The Pragmatic Bookshelf

Print Book/eBook: Practical Object-Oriented Design in Ruby

This isn’t specifically a testing book but I’ve seen it recommended a number of times as a book that will help you write better Ruby tests.

Excerpt from summary:
“[Practical Object-Oriented Design in Ruby] explains object-oriented design (OOD) using realistic, understandable examples. POODR* is a practical, readable introduction to how OOD can lower your costs and improve your applications.”

Details at Author’s Website

Section Two: Non-Rails-Specific Resources

Print Book: Growing Object-Oriented Software, Guided by Tests

This book is a classic in the testing world. I first read Growing Object-Oriented Software, Guided by Tests (GOOS) when I was clueless about testing. It helped me get oriented and learn what’s what. Among the most important concepts I learned from this book is the idea of a Walking Skeleton.

Details at Amazon

Print Book: Working Effectively with Legacy Code

I’ve worked on dozens of codebases so far in my career. Most of them have been legacy code. This book was super helpful in showing techniques like the Sprout Method technique to help get legacy code under control.

Details at Amazon

All code eventually gets tested, it’s just a question of when and how and by whom

Leave a reply

All code eventually gets tested. If a bug is found, it can be found in one of the following ways, in descending order of cost and embarrassment.

A Customer Trying to Use the Feature in Production

Obviously the worst place to find a bug is in production, and the worst person to find it is an end user trying to do his or her job.

This is a bad way to find a bug not just because it blocks your user from using your feature successfully and thus hurts your reputation, but because you might not even immediately find out that the bug exists. Not every user who finds a bug immediately reports that bug to the vendor’s development team. When I find a bug in Gmail, for example, I don’t make an attempt to report the bug to the Gmail team, I just suffer through it.

Even if the customer reports the bug, the bug probably won’t get directly reported to you, the developer. It will probably get reported to a support person, then get communicated to your manager, then communicated to you. By this point the bug may gain high visibility inside your organization, providing embarrassment to you and your team.

Discovering bugs in production is of course inevitable but all possible measures should be taken to avoid it.

Your Boss/Client Trying to Use It

The next worst way to find a bug is for your boss or client to find it.

I don’t mean when your boss/client serves in a QA tester role. I mean when your boss/client tries to use the feature with the expectation that it’s already finished and verified by you to be working.

The reason that this is bad is obvious: it erodes the trust your boss or client has in your competence as a developer. It especially hurts to get back a comment like, “Hey, let me know when this is 100% working.”

Not only is there the embarrassment and trust erosion but a concrete cost as well. The back-and-forth between you and your boss/client lengthens the development cycle for the feature, and therefore increases the cost to develop that feature.

Manually Testing it Yourself

Catching a bug by manually testing the feature yourself is way better than letting your boss, client or end user find it.

If you discover a silly bug, you might still feel a little embarrassed, but it will be a small, private embarrassment. You also keep the development cycle shorter by skipping the back-and-forth of “Hey boss, this is done,” “Hey developer, it’s not actually done.”

An Automated Test

The least expensive way to catch a bug is to write an automated test that makes the bug physically impossible.

Some developers perceive that writing automated tests takes “extra” time. That’s an inaccurate way to look at it. All testing takes time. The difference is that with a manual test, the time is paid each time you carry out the test, and with automated testing the time is paid all at once upfront. It’s like the difference between paying cash for a $250,000 house versus getting a mortgage and paying $500,000 for the same house over the course of 30 years. It potentially only takes a small handful of runs of a manual test before you’ve used up more time than it would have taken to write an automated test.

“So You’re Saying Automated Tests Are The Answer?”

My point is not that automated tests are a better alternative to all the other forms of testing. My point is that I believe automated tests are, in most cases, the least expensive testing method and the best first line of defense.

Imagine a secret government research facility that’s protected by an armed guard, locking doors and safe containing sensitive documents. If a Russian spy is after the documents, it’s obviously better for the spy to get caught by the guard than for the spy to make it past the guard and locked doors and make it all the way to the safe. But that doesn’t mean it’s a smart idea to do away with the doors and safe and only have the guard. Defense should exist at all levels practicable.

Same thing with automated testing. Just because it’s cheaper for a bug to get caught by an automated test doesn’t mean it’s a good idea to do away with manual testing. Protection should exist at all levels. It’s just a good idea to try to catch as many bugs at the earliest levels as possible.

How to Get RSpec to Skip View Specs When You Generate Scaffolds

Leave a reply

I personally don’t find much value in tests for views, helpers, routes or requests in most cases.

It’s annoying to have to delete these files each time you generate a new scaffold. Fortunately, it’s possible to configure RSpec not to generate these files.

Below is an example of how you can exclude these types of spec (and more) from being generated when you generate a new scaffold.

require_relative 'boot'

require "rails"
# Pick the frameworks you want:
require "active_model/railtie"
require "active_job/railtie"
require "active_record/railtie"
require "active_storage/engine"
require "action_controller/railtie"
require "action_mailer/railtie"
require "action_view/railtie"
require "action_cable/engine"
require "sprockets/railtie"
# require "rails/test_unit/railtie"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module MyApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 5.2

    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration can go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded after loading
    # the framework and any gems in your application.

    # Don't generate system test files.
    config.generators.system_tests = nil

    config.generators do |g|
      g.test_framework :rspec,
        fixtures:         false,
        view_specs:       false,
        helper_specs:     false,
        routing_specs:    false,
        request_specs:    false,
        controller_specs: false
    end
  end
end

Ruby/Rails testing glossary

1 Reply

Ruby-Specific Terms

Capybara
Integration testing tool. Capybara’s GitHub page describes it as an “acceptance test framework for web applications” but everyone I know uses the term integration test when they talk about Capybara.
Cucumber
A tool that lets you write test scenarios in English commands and then tie those English commands to code. For example, a Cucumber file could contain the line `Given I am on the login page`. In a different file, the text `Given I am on the login page` would map to some Ruby code that navigates to the login page.

I personally advise people not to use Cucumber.

Database Cleaner
The tests in a test suite should be runnable in any order. If a test leaves behind extra data or depends on data set up in a previous test, there’s a good chance the ability to run the tests in any order has been lost. Database Cleaner is a tool that helps ensure every test case starts and ends with a blank database.
factory_bot
“A library for setting up Ruby objects as test data”, according to factory_bot’s GitHub page. I think that’s a pretty good description. If I were to say it in my own words, I’d describe factory_bot as a library that helps define factories for the purpose of conveniently creating test data.
Faker
Faker is, for my purposes, a tool that generates random values so I don’t have to manually come up with arbitrary phone numbers, names of people, etc. when testing. Faker is also handy when I need test values to be unique, e.g. when having two people with the same name would be a problem. I use Faker in conjunction with factory_bot.
MiniTest
A Ruby testing framework.
RSpec
A Ruby testing framework. In my perception, RSpec is by far the most popular testing framework in the Rails community.
System Tests
A feature added to Rails core in Rails 5.1 to make integration testing easier. Uses Capybara. If you’d like to learn more about System Tests, Noel Rappin wrote a pretty good Rails System Tests guide.
Test::Unit
A Ruby testing framework.
timecop
A library that makes it easy to test time-dependent code. Offers features like “time travel” and “time freezing”.

Non-Ruby-Specific Terms

Acceptance Test
A test that checks whether a feature meets its business requirements. A certain feature may work exactly as the developer and designer intended, but if the developer and designer designed and built the wrong thing, the acceptance test would fail.
Assertion
A condition that evaluates to true if the feature behaves properly and false if the feature contains a bug. The analogous term in RSpec is “expectation”.
CI Server
A server that aids in the practice of continuous integration. A CI server typically runs the test suite anytime a code change is pushed and notifies the development team in the case of a test suite failure.
CircleCI
A CI server product.
Continuous Integration
The practice of continuously merging together the work of all the developers on a development team. The benefit of continuous integration is that it helps teams avoid the painful process of integrating large chunks of work, which is often tedious and error-prone.
Dependency Injection
A technique where an object’s dependencies are passed as arguments, making it easier to test objects in isolation. See this post of mine for an example.

Factory
There’s a design pattern called the Factory Pattern or the Factory Method Pattern and it exists independently of anything to do with tests. I basically think of a factory as code that spits out an object of some sort.

I included the Factory term here because factories are very relevant to how I write tests. Rather than manually spinning up a bunch of arbitrary data using fixtures, I prefer to define factories that can hide away irrelevant details, allowing me to focus only on the parts I care about. For example, if I want to create a restaurant for testing purposes, I might not want to have to specify the restaurant’s phone, address, hours, or any of those other details, even though they may be required attributes. Most of the time I just want to specify the restaurant’s name and nothing else. Factories can make this job very easy and convenient.

Fixture
A way of setting up test data. In Rails, fixtures are defined using YAML files. I prefer factories over fixtures for a number of reasons. One of the main reasons is that I find fixtures to be too “distant” from the tests, making it somewhat mysterious what data is available for any particular test.
Flapping test
A test that sometimes fails and sometimes passes, seemingly at random. Also known as a “flickering” test. They’re often a symptom of one test case “leaking” into subsequent test cases and affecting their behavior.
Edge Case
A path through a UI that isn’t exercised frequently or that is unlikely to be exercised.
End-to-End Test
A test that exercises all the layers of an application stack. It could be said that an end-to-end test is more inclusive than an integration test and that an integration test only has to combine two or more layers of an application stack to be considered an integration test. In practice I’ve found that people use the terms “integration test” and “end-to-end test” fairly interchangeably and if you want total clarity, you have to ask what exactly the person means.
Environment Parity
Environment parity is a broader version of the dev/prod parity term from the Twelve-Factor App principles. The idea is that every difference between two environments is an opportunity for a bug to be present in one environment that is not present in the other. For this reason it’s helpful to keep the test, development and production environments as similar to each other as possible.
Happy Path
A normal/default/valid path through a feature. (I like to think of the opposite as the “sad path”.)
Integration Test
A test that combines two or more layers of an application stack, or that combines two or more systems. See also: end-to-end test.
Mock
A “fake” object. A mock is useful when you want to test a certain object, but don’t want to deal with the hassle of bringing that object’s dependencies into the picture. The dependencies can instead be mocked.
Regression Testing
A type of testing meant to ensure that previously-working functionality has not been broken by newly-developed functionality.
Selenium
A tool for automating browsers. Also, a chemical element.
Stub
A term that may or may not mean the exact same thing as mock. Mock and stub are not terms that seem to have crisp industry consensus.
System Under Test
The system that’s being tested.
Test Case
I don’t think I can improve on Wikipedia‘s definition: “A specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective”.
Test Coverage
The percentage of an application covered by automated tests.
Test Environment
An independent instance of an application used specifically for the purpose of testing.
Test Suite
A collection of tests. An application might have one or many (or zero!) test suites.
Test-Driven Development
The practice of developing features by first writing tests that specify the behavior of the desired functionality, then writing the code to make those tests pass.
Unit Test
A fine-grained test that tests a very specific piece of functionality in isolation. Note: what Rails developers often refer to “unit tests” are not technically unit tests since those tests involve the database.