Author Archives: Jason Swett

Deploy to production on day one

Sleeping at the office, nothing to show

You know a project is under serious pressure when people start camping out at the office.

In 2015 I got a job at an agency where they were doing a big project for a large Chinese electronics manufacturing company. The project had been underway for about six months.

Two of the developers, who I’ll call Dave and Eric, had been working on the iOS and Android app portion of the project. I was told that for a time they had been sleeping at the office at night in an effort to hit their timelines.

The application had a Rails backend. I have a vague memory that the person who wrote most of the Rails app wasn’t around anymore.

Six months into the project, with a good portion of the budget already spent, the agency had no working application to show their client.

There were a lot of problems with the way the project had been carried out up to the point I got involved. They gave the design work to a very junior designer. The designs didn’t make sense and were not implementable. In the apps that were built, many things were only partway done and very little was all the way done.

There was also no production environment. For whatever reason it was decided that the project would be hosted on AWS. I had never touched AWS at the time but simply because I was the least unqualified person on staff to do it, the task of getting the Rails app onto AWS fell to me.

It took me forever to get things working on AWS. But I finally got it. It was not a simple deployment. The nature of the application required servers in different regions of the world. Because AWS regions are independent, we had to devise a complex messaging system to keep data in sync across multiple regions. It was a nightmare. Compounding the nightmare was the fact that we were under intense time pressure. Everything we were doing was supposed to have been done weeks or months prior.

Done by Thanksgiving? “I think we can do it!”

One of my most vivid memories from this project was kind of a “go/no go” meeting with a group of high-up people at the agency. This meeting included myself, the agency president, several people with “VP” in their title (which was almost everybody at the agency) and a few others. There were maybe about 12 people in the room. At this point in time we had very little working in production and about six weeks before “pencils down” time. If we didn’t have the project done by Thanksgiving, the whole thing was for naught.

At this meeting we discussed what still needed to be done relative to the amount of time we had left. The scope was not very negotiable. My guess was that in order to get everything done in our scope we would need about six months. I remember that we went around the table at this meeting and had each person say whether they thought we could or could not hit our Thanksgiving deadline. “I think we can do it,” the first person said. “I think we can do it!” the next person said. Every single person in the meeting said they thought we could do it. Well I didn’t think we could do it. So I said, “No, I don’t think we can do it.” I could tell everybody was annoyed with me and thought I was being a stick in the mud. Six months of work when we only had six weeks left seemed “too bad to be true”. I agreed it sucked that we couldn’t get six months’ work of work done in six weeks. But the amount that it would suck to not get it done had no bearing on whether we could actually get it done.

$300,000+ down the drain

Turns out I wasn’t crazy. We got nowhere near getting the project done in time. The client spent (as far as I understand) multiple hundreds of thousands of dollars and got nothing.

How this sad story could have been avoided

Not every part of every software project is equally challenging. Some tasks, like creating a new CRUD interface in a Rails project, are really familiar and pretty trivial to implement. Others, like setting up an AWS environment across different regions and setting up a message queuing system so they can talk to each other, when you’ve never touched AWS in your life, are not so trivial.

When you have a mix of mysterious and non-mysterious work in a project you should do the most mysterious work first. Don’t save the super hard stuff for last.

You might think this is common sense. I would think so too. But surprisingly many people leave the most mysterious part of a project for the 11th hour.

This agency’s failed project might not have been quite such a spectacular failure if they had deployed to production on day one. If you deploy a mostly-empty application to production on day one, there’s way less stuff to have to get working. If something breaks in the process of setting up the production environment, you have way less stuff to sort through to figure out exactly where the problem lies. The stress level is also much lower at the beginning of a project.

Getting a production environment in place on day one makes it so every subsequent deployment is a non-event. Throughout the project you can do a number of deployments every week. If the project deadline is Thanksgiving there need not be a mad scramble to get things set up and working the day before Thanksgiving. You just deploy like normal. The workload might be heavier but the deployment process is uneventful.

For less risk and less stress, deploy to production on day one.

What does all this have to do with automated testing?

Like the authors of GOOS say, a healthy development process involves feedback loops ranging from seconds to months. These kinds of feedback loops may include:

  • Unit tests (seconds)
  • Entire unit + integration test suites (minutes)
  • Pair programming sessions (seconds to minutes)
  • PR reviews (minutes to hours)
  • Feature releases (days to weeks)
  • Entire development projects (weeks to months)

These feedback loops are enabled by practices like automated testing, pair programming, continuous integration and continuous delivery/deployment. These tests don’t exist in isolation but weave and synergize with each other. It’s hard to practice continuous deployment when you don’t have any tests. Continuous integration shines a light on your tests and forces you to pay attention to how healthy your test suite is. In addition to deploying to production on day one, it’s immensely helpful to get your testing infrastructure in place on day one and release a tiny feature complete with tests.

How I write model tests

I recently read a post on /r/ruby that asked for RSpec model testing best practices. I gave an answer, but also wanted to give some concrete examples, so here we go.

What follows is how I’d write a spec for a new model. I actually took a model from an existing application I wrote and just blew away the code so I could start over from scratch.

Once I got a ways into this post I realized that it was getting pretty long and I still hadn’t gotten very far into the “meat” of how I’d write a model spec. So I intend to write a follow-up post that goes more in-depth.

Starting point

Here’s what my `Restaurant` model and test look like when they’re empty.

class Restaurant < ApplicationRecord
end
require 'rails_helper'

RSpec.describe Restaurant, type: :model do
end

And just so you can see what the `Restaurant` class’s attributes are, here’s a snippet of `db/schema.rb`. Most of these attributes won’t come into the picture. We’ll mostly just deal with `name` and `phone`.

create_table "restaurants", force: :cascade do |t|
  t.string "name", null: false
  t.datetime "created_at", null: false
  t.datetime "updated_at", null: false
  t.string "phone", null: false
  t.integer "business_model_id", null: false
  t.index ["business_model_id"], name: "index_restaurants_on_business_model_id"
  t.index ["name"], name: "index_restaurants_on_name", unique: true
  t.index ["phone"], name: "index_restaurants_on_phone", unique: true
end

The first spec

To me the most natural thing to test is the presence of the restaurant name. I’ll write a failing test for presence of name using Should Matchers.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
end

When I run this spec, it fails, as I expect.

rspec spec/models/restaurant_spec.rb    
F                                  

Failures:                          

  1) Restaurant should validate that :name cannot be empty/falsy       
     Failure/Error: it { is_expected.to validate_presence_of(:name) }  
                                   
       Restaurant did not properly validate that :name cannot be empty/falsy.                                                                  
         After setting :name to ‹nil›, the matcher expected the Restaurant to                                                                  
         be invalid, but it was valid instead.                         
     # ./spec/models/restaurant_spec.rb:4:in `block (2 levels) in <top (required)>'                                                            

Finished in 0.29263 seconds (files took 1.34 seconds to load)          
1 example, 1 failure               

Failed examples:                   

rspec ./spec/models/restaurant_spec.rb:4 # Restaurant should validate that :name cannot be empty/falsy

I add the name validator to make the spec pass.

class Restaurant < ApplicationRecord
  validates :name, presence: true
end

And indeed, the spec now passes.

rspec spec/models/restaurant_spec.rb
.

Finished in 0.2793 seconds (files took 1.32 seconds to load)
1 example, 0 failures

Spec for phone presence

Now that name presence is taken care of I turn my attention to phone. The spec in this case is the exact same.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
  it { is_expected.to validate_presence_of(:phone) }
end

I run the spec and it fails. I then add a presence validator:

class Restaurant < ApplicationRecord
  belongs_to :business_model
  validates :name, presence: true
  validates :phone, presence: true
end

The spec passes now.

Phone number format validity

Unlike restaurant name, which could be pretty much anything, the phone number has to have a valid format. For example, “123” of course isn’t a valid phone number. I add a failing test for this case. I’m actually not sure what I expect the error message to be, so I just put “invalid format”. After I run the spec I can update the error message in my test to match the actual error message.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
  it { is_expected.to validate_presence_of(:phone) }

  describe 'when phone number is too short' do
    it 'is not valid' do
      restaurant = build(:restaurant, phone: '123')
      restaurant.valid?
      expect(restaurant.errors[:phone]).to include('invalid format')
    end
  end
end

As I expect, this test fails.

rspec spec/models/restaurant_spec.rb    
..F                                

Failures:                          

  1) Restaurant when phone number is too short is not valid            
     Failure/Error: expect(restaurant.errors[:phone]).to include('invalid format')                                                             
       expected [] to include "invalid format"                         
     # ./spec/models/restaurant_spec.rb:11:in `block (3 levels) in <top (required)>'                                                           

Finished in 0.33318 seconds (files took 1.31 seconds to load)          
3 examples, 1 failure              

Failed examples:                   

rspec ./spec/models/restaurant_spec.rb:8 # Restaurant when phone number is too short is not valid

Now I add a format validator using a regex I found on the internet.

class Restaurant < ApplicationRecord
  belongs_to :business_model
  validates :name, presence: true
  validates :phone, presence: true, format: {
    with: /\A(\+\d{1,2}\s)?\(?\d{3}\)?[\s.-]\d{3}[\s.-]\d{4}\z/
  }
end

The spec now fails because the expected error message, “invalid format” doesn’t match the actual error message, “is invalid”.

rspec spec/models/restaurant_spec.rb
..F

Failures:

  1) Restaurant when phone number is too short is not valid
     Failure/Error: expect(restaurant.errors[:phone]).to include('invalid format')
       expected ["is invalid"] to include "invalid format"
     # ./spec/models/restaurant_spec.rb:11:in `block (3 levels) in <top (required)>'

Finished in 0.40668 seconds (files took 2.03 seconds to load)
3 examples, 1 failure

Failed examples:

rspec ./spec/models/restaurant_spec.rb:8 # Restaurant when phone number is too short is not valid

So I update my expected error message.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
  it { is_expected.to validate_presence_of(:phone) }

  describe 'when phone number is too short' do
    it 'is not valid' do
      restaurant = build(:restaurant, phone: '123')
      restaurant.valid?
      expect(restaurant.errors[:phone]).to include('is invalid')
    end
  end
end

And now the spec passes.

rspec spec/models/restaurant_spec.rb    
...                                

Finished in 0.38825 seconds (files took 2.03 seconds to load)          
3 examples, 0 failures         

Tests for other phone cases

By adding this phone regex I’ve actually broken a rule of TDD: only write enough code to make the test pass. If you only write enough code to make the test pass, you know that all the code you’ve written is covered by tests.

In this case I “wrote some code” (copy/pasted a regex from Stack Overflow) that didn’t have tests. So I’m going to go back and add more test cases. One case will say, “when the phone number is valid, the restaurant object should be valid”. The other will say “when the phone number is all numbers, the restaurant object should not be valid”.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
  it { is_expected.to validate_presence_of(:phone) }

  describe 'phone' do
    describe 'when phone number is valid' do
      it 'is valid' do
        restaurant = build(:restaurant, phone: '(555) 555-5555')
        expect(restaurant).to be_valid
      end
    end

    describe 'when phone number is too short' do
      it 'is not valid' do
        restaurant = build(:restaurant, phone: '123')
        restaurant.valid?
        expect(restaurant.errors[:phone]).to include('is invalid')
      end
    end

    describe 'when phone number is all letters' do
      it 'is not valid' do
        restaurant = build(:restaurant, phone: '(AAA) AAA-AAAA')
        restaurant.valid?
        expect(restaurant.errors[:phone]).to include('is invalid')
      end
    end
  end
end

These tests all pass. There’s a little duplication in my test, though, so I’m going to refactor.

require 'rails_helper'

RSpec.describe Restaurant, type: :model do
  it { is_expected.to validate_presence_of(:name) }
  it { is_expected.to validate_presence_of(:phone) }

  describe 'phone' do
    let(:restaurant) { build(:restaurant) }

    describe 'when phone number is valid' do
      it 'is valid' do
        restaurant.phone = '(555) 555-5555'
        expect(restaurant).to be_valid
      end
    end

    describe 'when phone number is too short' do
      it 'is not valid' do
        restaurant.phone = '123'
        restaurant.valid?
        expect(restaurant.errors[:phone]).to include('is invalid')
      end
    end

    describe 'when phone number is all letters' do
      it 'is not valid' do
        restaurant.phone = '(AAA) AAA-AAAA'
        restaurant.valid?
        expect(restaurant.errors[:phone]).to include('is invalid')
      end
    end
  end
end

All the tests still pass.

rspec spec/models/restaurant_spec.rb    
.....                              

Finished in 0.30868 seconds (files took 1.37 seconds to load)          
5 examples, 0 failures

To be continued

What I’ve written in the post is representative of the start of what I’d put in a model test but it’s certainly not the whole thing. What if my model contains some non-trivial methods? This post is already getting kind of long, so I plan to continue this in a Part 2.

Things you can ignore when getting started with Rails testing

Here’s an incomplete list of tools and concepts you might encounter when first trying to learn Rails testing: Capybara, Cucumber, Database Cleaner, factory_bot, Faker, MiniTest, RSpec, system tests, Test::Unit, acceptance tests, end-to-end-tests, mocks, stubs, unit tests and TDD.

That’s a lot of stuff. If you’re like most humans you might look at this long list of things, feel confused about where to start, and say, “I don’t know what to do. I’ll just deal with this later.”

The challenge of getting started with testing would be much easier if you knew exactly what you needed to know and what you could safely ignore. What follows is a list of what you can safely ignore.

Framework Decisions

You don’t need to get hung up on which framework to use. You literally can’t go wrong. The principles of testing are the same no matter which testing framework you use. Plus you can always change your mind later.

When I first got started with Rails, the way I decided on a testing framework was very simple. I noticed that most Rails developers used RSpec, so I just picked RSpec.

(I actually used Test::Unit for a while before realizing most Rails developers used RSpec. So I just switched. It wasn’t a very big deal.)

Cucumber, Capybara and System Tests

Most Rails test suites have two main sub-suites: a suite of model tests and a suite of integration tests.

Model tests test the behavior of ActiveRecord models by themselves. Integration tests actually spin up a browser and do things like fill out forms and click links and buttons as if a human is actually using the application.

Between these two, the bar is lower for model tests. For integration tests you have to do almost all the same stuff as model tests plus more. For this reason I suggest that if you’re a newcomer to Rails testing that you start with model tests (or even just Ruby tests without Rails) and ignore integration tests altogether until you get more comfortable with testing.

So you can ignore Cucumber (which I don’t recommend using at all), Capybara and system tests, which are all integration testing tools.

View Specs

I’ve never written a view spec. To me they seem tautological. I’ve also never encountered any other Rails developer who advocates writing view specs.

If for some reason I had a wildly complicated view, I could see a view spec potentially making sense. But I haven’t yet encountered that case.

Helper Specs

I don’t tend to write helper specs because I don’t tend to write helpers. Helpers are a fairly peripheral area of Rails that you can safely disregard altogether when you’re getting started with Rails testing.

Routing Specs

Like view specs, I find routing specs to be tautological. I don’t write them.

Request Specs

Request specs are a great boon and even necessity if your application has an API or if your controllers do anything non-trivial. But I don’t think you should worry about request specs when you’re just getting started.

Controller Specs

Controller specs are deprecated in favor of request specs.

What Not to Ignore

The main thing I would recommend learning when you’re getting started with Rails testing is model tests.

When you’re learning about model tests you’ll naturally have to get acquainted with RSpec syntax (or whichever framework you choose), factory_bot and Database Cleaner (or analogous tools). But other than the actual testing techniques, that’s about it as far as model test tooling goes.

If you want to make life even easier on yourself you can learn just Ruby and RSpec with no Rails involved. Then, after you get comfortable with RSpec syntax and basic testing techniques, you can approach Rails testing with more confidence.

Why I recommend against using Cucumber

Around the time I first started using Rails in 2011, I noticed that a lot of developers, seemingly all Rails developers, were using Cucumber to assist with testing.

I bought into the idea—describing test cases in plain English—but in practice I found Cucumber not to be very valuable. In fact, my experience has been that Cucumber adds a negative amount of value.

In recent years I’ve noticed (although this is anecdotal and might just be my perception) that fewer codebases seem to use Cucumber and that fewer Rails developers seem to be on board with Cucumber. I had thought Cucumber was pretty much dead. But lately, to my surpise, I’ve seen Cucumber recommended to testing noobs more than a few times. Since I consider Cucumber to be a bad thing, I want to explain why I think so and why I don’t think other people should use it.

In my view there are two general ways Cucumber can be used: it can be used as intended or it can be abused. In the former case, I believe Cucumber has a small negative value. In the latter case I believe it has a large negative value.

Why Cucumber is bad when it’s not used as intended

Most production Cucumber scenarios I’ve seen look something like this:

Given a user exists with email "test@example.com" and password "mypassword"
And I visit "/sign_in"
And I fill in the "Email" field with "test@example.com"
And I fill in the "Password" field with "mypassword"
And I click "Sign In"
And I visit "/user/edit"
And I fill in the "First Name" field with "John"
And I fill in the "Last Name" field with "Smith"
And I fill in the "Age" field with "30"
And I click "Save"
And I visit "/profile"
Then I should see "John Smith, 30"

These kinds of tests, with fine-grained steps, arise when the developers seem to mistake Cucumber for a way to write Ruby in English. The above scenario provides exactly zero benefit, in my opinion, over the following equivalent Capybara scenario:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The Cucumber/Gherkin version is no shorter nor more easily understandable.

To be fair to Cucumber, nobody who understands Cucumber advocates writing Cucumber scenarios in this way. The Cucumber creator himself, Aslak Hellesøy, wrote a post in 2011 saying not to do this. Other people have written similar things.

I think it’s telling that so many people have written blog posts advising against the very common practice of writing fine-grained Cucumber steps. To me it’s kind one of those gas station doors that looks for all the world like a pull door, so every single person who comes up to it pulls it instead of pushes it, feels like a dumbass, and then pushes it. So the gas station manager puts up a big sign that says “PUSH”, but most people don’t notice it and the problem persists. What instead should have been done is to make the push door look like a push door, without the big useless handle that you’re not supposed to pull. I get that the Cucumber maintainers tried to do that by removing `web_steps.rb`, but in my experience it didn’t seem to work.

And it doesn’t matter much anyway because Cucumber still sucks even if you don’t abuse it by writing fine-grained steps. I’ll explain why I think so.

Why Cucumber is bad even when it is used as intended

Here’s a version of the above Cucumber scenario that’s done in the way the Cucumber creators would intend. There are two parts.

First, the Gherkin steps:

Given I am signed in
And I provide my name and age details
Then I should see those details on my profile page

Second, the underlying Ruby steps:

Given /^I am signed in$/ do
  visit sign_in_path
  fill_in 'Email', with: 'test@example.com'
  fill_in 'Password', with: 'mypassword'
  click_on 'Sign In'
end

And /^I provide my name and age details$/ do
  visit edit_user_path
  fill_in 'First Name', with: 'John'
  fill_in 'Last Name', with: 'Smith'
  fill_in 'Age', with: '30'
  click_on 'Save'
end

Then /^I should see those details on my profile page$/ do
  visit profile_path
  expect(page).to have_content('John Smith, 30')
end

This is actually pretty decent-looking and appealing, at least at first glance. There are two problems, though. First, this way of doing things doesn’t really provide any clarity over doing it the Capybara way. Second, the step definitions usually end up in a single, flat file full of “step soup” where unrelated steps are mixed together willy-nilly.

Compare this again with the Capybara version:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The sign in portion is usually abstracted away in Capybara, too, so the scenario would look more like this:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

sign_in

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

That’s not too crazy at all. In order for Cucumber to be a superior solution to using bare Capybara, it would have to have some pretty strong benefits to compensate for the maintenance burden and cognitive overhead it adds. But it doesn’t.

So what do I recommend doing instead of using Cucumber? I think just using Capybara by itself is fine, and better than using Capybara + Cucumber. I also think Capybara + page objects is a pretty good way to go.

[xyz-ihs snippet=”rtmc-opt-in”]

Rails testing resource roundup

Below is a list of testing resources I’ve either used myself or heard recommended. I tried to make the list as Ruby-centric as possible, although some resources from other ecosystems are so good that I didn’t want to exclude them.

I intend this to be a living list that grows over time. If you know of something that should be on this list but isn’t, please let me know, either in a comment on this page or via email.

Disclosure statement: None of the links below are affiliate links, although I do have a relationship with some of the authors/creators of these resources.

Section One: Ruby/Rails Specific Resources

Print Book: Rails 5 Test Prescriptions


Excerpt from Pragmatic Bookshelf summary:

“Does your Rails code suffer from bloat, brittleness, or inaccuracy? Cure these problems with the regular application of test-driven development. You’ll use Rails 5.2, Minitest 5, and RSpec 3.7, as well as popular testing libraries such as factory_bot and Cucumber.”

Details at The Pragmatic Bookshelf

Side note: you can also listen to my interview with author Noel Rappin on the Ruby Testing Podcast.

eBook: Everyday Rails Testing with RSpec


Summary from Leanpub:

“Real-world advice for adding reliable tests to your Rails apps with RSpec, complete with expanded, exclusive content and a full sample application. Updates for 2017 now available—RSpec 3.6, Rails 5.1, and more! Learn to test with confidence!”

Details at Leanpub

Screencast Series: Destroy All Software

Not everything on Destroy All Software (DAS) is testing-related but a lot of it is. I often see DAS recommended when people ask for testing-related resources.

Destroy All Software Catalog

Online Course Series: Upcase’s “Learn Testing” Courses

Summary from Upcase website:

“Test-driven development, or TDD, is the practice of writing your tests firsts, then using those tests to guide you as you write your actual production code. This may sound crazy, but it turns out that it makes writing code much easier. It provides a clear workflow and next steps while you’re building and has the added benefit of producing a test suite you can have confidence in. With these courses and videos we’ll teach you everything you need to know to get started with TDD.”

Details at Upcase

Print Book/eBook: Effective Testing with RSpec 3

Excerpt from Pragmatic Bookshelf summary:

“This definitive guide from RSpec’s lead developer shows you how to use RSpec to drive more maintainable designs, specify and document expected behavior, and prevent regressions during refactoring. Build a project using RSpec to design, describe, and test the behavior of your code. Whether you’re new to automated tests or have been using them for years, this book will help you write more effective tests.”

Details at The Pragmatic Bookshelf

Print Book/eBook: Practical Object-Oriented Design in Ruby

This isn’t specifically a testing book but I’ve seen it recommended a number of times as a book that will help you write better Ruby tests.

Excerpt from summary:
“[Practical Object-Oriented Design in Ruby] explains object-oriented design (OOD) using realistic, understandable examples. POODR* is a practical, readable introduction to how OOD can lower your costs and improve your applications.”

Details at Author’s Website

Section Two: Non-Rails-Specific Resources

Print Book: Growing Object-Oriented Software, Guided by Tests

This book is a classic in the testing world. I first read Growing Object-Oriented Software, Guided by Tests (GOOS) when I was clueless about testing. It helped me get oriented and learn what’s what. Among the most important concepts I learned from this book is the idea of a Walking Skeleton.

Details at Amazon

Print Book: Working Effectively with Legacy Code

I’ve worked on dozens of codebases so far in my career. Most of them have been legacy code. This book was super helpful in showing techniques like the Sprout Method technique to help get legacy code under control.

Details at Amazon

All code eventually gets tested, it’s just a question of when and how and by whom

All code eventually gets tested. If a bug is found, it can be found in one of the following ways, in descending order of cost and embarrassment.

A Customer Trying to Use the Feature in Production

Obviously the worst place to find a bug is in production, and the worst person to find it is an end user trying to do his or her job.

This is a bad way to find a bug not just because it blocks your user from using your feature successfully and thus hurts your reputation, but because you might not even immediately find out that the bug exists. Not every user who finds a bug immediately reports that bug to the vendor’s development team. When I find a bug in Gmail, for example, I don’t make an attempt to report the bug to the Gmail team, I just suffer through it.

Even if the customer reports the bug, the bug probably won’t get directly reported to you, the developer. It will probably get reported to a support person, then get communicated to your manager, then communicated to you. By this point the bug may gain high visibility inside your organization, providing embarrassment to you and your team.

Discovering bugs in production is of course inevitable but all possible measures should be taken to avoid it.

Your Boss/Client Trying to Use It

The next worst way to find a bug is for your boss or client to find it.

I don’t mean when your boss/client serves in a QA tester role. I mean when your boss/client tries to use the feature with the expectation that it’s already finished and verified by you to be working.

The reason that this is bad is obvious: it erodes the trust your boss or client has in your competence as a developer. It especially hurts to get back a comment like, “Hey, let me know when this is 100% working.”

Not only is there the embarrassment and trust erosion but a concrete cost as well. The back-and-forth between you and your boss/client lengthens the development cycle for the feature, and therefore increases the cost to develop that feature.

Manually Testing it Yourself

Catching a bug by manually testing the feature yourself is way better than letting your boss, client or end user find it.

If you discover a silly bug, you might still feel a little embarrassed, but it will be a small, private embarrassment. You also keep the development cycle shorter by skipping the back-and-forth of “Hey boss, this is done,” “Hey developer, it’s not actually done.”

An Automated Test

The least expensive way to catch a bug is to write an automated test that makes the bug physically impossible.

Some developers perceive that writing automated tests takes “extra” time. That’s an inaccurate way to look at it. All testing takes time. The difference is that with a manual test, the time is paid each time you carry out the test, and with automated testing the time is paid all at once upfront. It’s like the difference between paying cash for a $250,000 house versus getting a mortgage and paying $500,000 for the same house over the course of 30 years. It potentially only takes a small handful of runs of a manual test before you’ve used up more time than it would have taken to write an automated test.

“So You’re Saying Automated Tests Are The Answer?”

My point is not that automated tests are a better alternative to all the other forms of testing. My point is that I believe automated tests are, in most cases, the least expensive testing method and the best first line of defense.

Imagine a secret government research facility that’s protected by an armed guard, locking doors and safe containing sensitive documents. If a Russian spy is after the documents, it’s obviously better for the spy to get caught by the guard than for the spy to make it past the guard and locked doors and make it all the way to the safe. But that doesn’t mean it’s a smart idea to do away with the doors and safe and only have the guard. Defense should exist at all levels practicable.

Same thing with automated testing. Just because it’s cheaper for a bug to get caught by an automated test doesn’t mean it’s a good idea to do away with manual testing. Protection should exist at all levels. It’s just a good idea to try to catch as many bugs at the earliest levels as possible.

How to Get RSpec to Skip View Specs When You Generate Scaffolds

I personally don’t find much value in tests for views, helpers, routes or requests in most cases.

It’s annoying to have to delete these files each time you generate a new scaffold. Fortunately, it’s possible to configure RSpec not to generate these files.

Below is an example of how you can exclude these types of spec (and more) from being generated when you generate a new scaffold.

require_relative 'boot'

require "rails"
# Pick the frameworks you want:
require "active_model/railtie"
require "active_job/railtie"
require "active_record/railtie"
require "active_storage/engine"
require "action_controller/railtie"
require "action_mailer/railtie"
require "action_view/railtie"
require "action_cable/engine"
require "sprockets/railtie"
# require "rails/test_unit/railtie"

# Require the gems listed in Gemfile, including any gems
# you've limited to :test, :development, or :production.
Bundler.require(*Rails.groups)

module MyApp
  class Application < Rails::Application
    # Initialize configuration defaults for originally generated Rails version.
    config.load_defaults 5.2

    # Settings in config/environments/* take precedence over those specified here.
    # Application configuration can go into files in config/initializers
    # -- all .rb files in that directory are automatically loaded after loading
    # the framework and any gems in your application.

    # Don't generate system test files.
    config.generators.system_tests = nil

    config.generators do |g|
      g.test_framework :rspec,
        fixtures:         false,
        view_specs:       false,
        helper_specs:     false,
        routing_specs:    false,
        request_specs:    false,
        controller_specs: false
    end
  end
end

Ruby/Rails testing glossary

Ruby-Specific Terms

Capybara
Integration testing tool. Capybara’s GitHub page describes it as an “acceptance test framework for web applications” but everyone I know uses the term integration test when they talk about Capybara.
Cucumber
A tool that lets you write test scenarios in English commands and then tie those English commands to code. For example, a Cucumber file could contain the line `Given I am on the login page`. In a different file, the text `Given I am on the login page` would map to some Ruby code that navigates to the login page.

I personally advise people not to use Cucumber.

Database Cleaner
The tests in a test suite should be runnable in any order. If a test leaves behind extra data or depends on data set up in a previous test, there’s a good chance the ability to run the tests in any order has been lost. Database Cleaner is a tool that helps ensure every test case starts and ends with a blank database.
factory_bot
“A library for setting up Ruby objects as test data”, according to factory_bot’s GitHub page. I think that’s a pretty good description. If I were to say it in my own words, I’d describe factory_bot as a library that helps define factories for the purpose of conveniently creating test data.
Faker
Faker is, for my purposes, a tool that generates random values so I don’t have to manually come up with arbitrary phone numbers, names of people, etc. when testing. Faker is also handy when I need test values to be unique, e.g. when having two people with the same name would be a problem. I use Faker in conjunction with factory_bot.
MiniTest
A Ruby testing framework.
RSpec
A Ruby testing framework. In my perception, RSpec is by far the most popular testing framework in the Rails community.
System Tests
A feature added to Rails core in Rails 5.1 to make integration testing easier. Uses Capybara. If you’d like to learn more about System Tests, Noel Rappin wrote a pretty good Rails System Tests guide.
Test::Unit
A Ruby testing framework.
timecop
A library that makes it easy to test time-dependent code. Offers features like “time travel” and “time freezing”.

Non-Ruby-Specific Terms

Acceptance Test
A test that checks whether a feature meets its business requirements. A certain feature may work exactly as the developer and designer intended, but if the developer and designer designed and built the wrong thing, the acceptance test would fail.
Assertion
A condition that evaluates to true if the feature behaves properly and false if the feature contains a bug. The analogous term in RSpec is “expectation”.
CI Server
A server that aids in the practice of continuous integration. A CI server typically runs the test suite anytime a code change is pushed and notifies the development team in the case of a test suite failure.
CircleCI
A CI server product.
Continuous Integration
The practice of continuously merging together the work of all the developers on a development team. The benefit of continuous integration is that it helps teams avoid the painful process of integrating large chunks of work, which is often tedious and error-prone.
Dependency Injection
A technique where an object’s dependencies are passed as arguments, making it easier to test objects in isolation. See this post of mine for an example.

Factory
There’s a design pattern called the Factory Pattern or the Factory Method Pattern and it exists independently of anything to do with tests. I basically think of a factory as code that spits out an object of some sort.

I included the Factory term here because factories are very relevant to how I write tests. Rather than manually spinning up a bunch of arbitrary data using fixtures, I prefer to define factories that can hide away irrelevant details, allowing me to focus only on the parts I care about. For example, if I want to create a restaurant for testing purposes, I might not want to have to specify the restaurant’s phone, address, hours, or any of those other details, even though they may be required attributes. Most of the time I just want to specify the restaurant’s name and nothing else. Factories can make this job very easy and convenient.

Fixture
A way of setting up test data. In Rails, fixtures are defined using YAML files. I prefer factories over fixtures for a number of reasons. One of the main reasons is that I find fixtures to be too “distant” from the tests, making it somewhat mysterious what data is available for any particular test.
Flapping test
A test that sometimes fails and sometimes passes, seemingly at random. Also known as a “flickering” test. They’re often a symptom of one test case “leaking” into subsequent test cases and affecting their behavior.
Edge Case
A path through a UI that isn’t exercised frequently or that is unlikely to be exercised.
End-to-End Test
A test that exercises all the layers of an application stack. It could be said that an end-to-end test is more inclusive than an integration test and that an integration test only has to combine two or more layers of an application stack to be considered an integration test. In practice I’ve found that people use the terms “integration test” and “end-to-end test” fairly interchangeably and if you want total clarity, you have to ask what exactly the person means.
Environment Parity
Environment parity is a broader version of the dev/prod parity term from the Twelve-Factor App principles. The idea is that every difference between two environments is an opportunity for a bug to be present in one environment that is not present in the other. For this reason it’s helpful to keep the test, development and production environments as similar to each other as possible.
Happy Path
A normal/default/valid path through a feature. (I like to think of the opposite as the “sad path”.)
Integration Test
A test that combines two or more layers of an application stack, or that combines two or more systems. See also: end-to-end test.
Mock
A “fake” object. A mock is useful when you want to test a certain object, but don’t want to deal with the hassle of bringing that object’s dependencies into the picture. The dependencies can instead be mocked.
Regression Testing
A type of testing meant to ensure that previously-working functionality has not been broken by newly-developed functionality.
Selenium
A tool for automating browsers. Also, a chemical element.
Stub
A term that may or may not mean the exact same thing as mock. Mock and stub are not terms that seem to have crisp industry consensus.
System Under Test
The system that’s being tested.
Test Case
I don’t think I can improve on Wikipedia‘s definition: “A specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective”.
Test Coverage
The percentage of an application covered by automated tests.
Test Environment
An independent instance of an application used specifically for the purpose of testing.
Test Suite
A collection of tests. An application might have one or many (or zero!) test suites.
Test-Driven Development
The practice of developing features by first writing tests that specify the behavior of the desired functionality, then writing the code to make those tests pass.
Unit Test
A fine-grained test that tests a very specific piece of functionality in isolation. Note: what Rails developers often refer to “unit tests” are not technically unit tests since those tests involve the database.