Category Archives: Automated Testing

What should I NOT write tests for?

One of the most common questions about testing, including what to write tests for, is what NOT to write tests for.

When people ask me what to write tests for, my honest but maybe not very helpful answer is “basically everything”. But I don’t test literally absolutely everything. There are some cases when I choose to skip tests.

My criteria for skipping a test

My habit of writing tests for everything leads me to write tests for my code by default. For most of the code I write, I actually finder it harder to write the code without tests than with tests.

But sometimes I get lazy or my instincts tell me that a test would be a waste of time. When I’m feeling like this, I ask myself three questions:

  1. If this code were to misbehave, would it fail silently?
  2. If this code were to misbehave, how bad would the consequences be?
  3. If this code were to misbehave, how frequently would it fail?
  4. How costly would it be to write a test?

If the code I’m working on would fail in a very obvious way, and the consequences are minor, and the failure would only happen once a year, and the test would be very costly to write, then that’s a case where I would probably skip the test.

On the other hand, if the code would fail silently, OR if the consequences would be bad, OR if the failure would be frequent, OR if it’s super easy to write the test, then I would just write the test.

You could think of these questions as a boolean expression where all the items get OR’d together:

“Should I write a test?” formula (boolean OR)

  1. Might fail silently?
  2. Consequences might be bad?
  3. Might fail frequently?
  4. Test is easy to write?

If any one of the items is true, then the whole boolean expression is true and I go ahead and write the test. Otherwise I skip the test without guilt or worry.

Behavior vs. implementation

There are also entire types of tests I avoid.

I don’t test for things like the presence of associations, the presence of methods, etc. Such tests are pointless. Rather, I test the behavior that these things enable. Testing the behavior is the only way you can really be sure anything works.

Takeaways

  • I’ll skip a test if and only if the feature won’t fail silently, the consequences won’t be bad, the failure won’t occur frequently, and the test is expensive to write.
  • I don’t test implementation details, but rather I test the behaviors that the implementations enable.

Understanding Factory Bot syntax by coding your own Factory Bot

What we’re going to do and why

When you look at a Factory Bot factory definition, the syntax might look somewhat mysterious. Here’s an example of such a factory.

FactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

The goal of this tutorial is to demystify this syntax. The way we’ll do this is to write our own implementation of Factory Bot from scratch. Or, more precisely, we’ll write an implementation of something that behaves indistinguishably from Factory Bot for a few narrow use cases.

Concepts we’ll learn about

Blocks

Factory Bot syntax makes heavy use of blocks. In this post we’ll learn a little bit about how blocks work.

Message sending

We’ll learn the nuanced distinction between calling a method on an object and sending a message to an object.

method_missing

We’ll learn how to use Ruby’s method_missing feature so that we can define methods for objects dynamically.

Our objective

Our goal with this post will be to write some code that makes the factory below actually work. Notice that this factory is indistinguishable from a Factory Bot factory except for the fact that it starts with MyFactoryBot rather than FactoryBot.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

In addition to this factory code, we’ll also need some code that exercises the factory.

Exercising the factory

Here’s some code that will exercise our factory to make sure it actually works. Just like how our factory definition mirrors an actual Factory Bot factory definition, the below code mirrors how we would use a Factory Bot factory.

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

After we get our factory working properly, the above code should produce the following output.

First name: John
Last name: Smith

Let’s get started.

How to follow along

If you’d like to code along with this tutorial, I have a sample project that you can use.

The code in this tutorial depends on a certain Rails project, so I’ve created a GitHub repo at https://github.com/jasonswett/my_factory_bot where there’s a Dockerized Rails app, the same exact Rails app that I used to write this tutorial. You’ll find instructions to set up the project on the GitHub page.

The approach

We’ll be writing our “Factory Bot clone” code using a (silly) development methodology that I call “error-driven development”. Error-driven development works like this: you write a piece of code and try to run it. If you get an error when you run the code, you write just enough code to fix that particular error, and nothing more. You repeat this process until you have the result you want.

The reason I sometimes like to code this way is that it prevents me from writing any code that hasn’t been (manually) tested. Surprisingly enough, this “methodology” actually works pretty well.

Building the factory

The first thing to do is to create a file called my_factory_bot.rb and put it at the Rails project root.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Then we’ll run the code like this:

$ rails run my_factory_bot.rb

The first thing we’ll see is an error saying that MyFactoryBot is not defined.

uninitialized constant MyFactoryBot (NameError)

This is of course true. We haven’t yet defined something called MyFactoryBot. So, in the spirit of practicing “error-driven development”, let’s write enough code to make this particular error go away and nothing more.

class MyFactoryBot
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now, if we run the code again (using the same rails run command from above), we get a different error.

undefined method `define' for MyFactoryBot:Class (NoMethodError)

This is also true. The MyFactoryBot class doesn’t have a method called define. So let’s define it.

class MyFactoryBot
  def self.define
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we get a new error.

undefined method `create' for MyFactoryBot:Class (NoMethodError)

This of course comes from the user = MyFactoryBot.create(:user) line. Let’s define a create method in order to make this error go away. Since we’re passing in an argument, :user, when we call create, we’ll need to specify a parameter for the create method. I’m calling the parameter model_sym since it’s a symbol that corresponds to the model that the factory is targeting.

class MyFactoryBot
  def self.define
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we get an error for the next line.

undefined method `first_name' for nil:NilClass (NoMethodError)

We’ll deal with this error, but not just yet, because this one will be a little bit tricky, and there are some certain other things that will make sense to do first. Let’s temporarily comment out the lines that call first_name and last_name.

class MyFactoryBot
  def self.define
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now, if we run the file again, we get no errors.

Making it so the factory block gets called

Right now, the block inside of MyFactoryBot.define isn’t getting used at all. Let’s add block.call to the defined method so that the block gets called.

class MyFactoryBot
  def self.define(&block)
    block.call
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now when we run the file we get the following error.

undefined method `factory' for main:Object (NoMethodError)

It makes sense that we would get an error that says “undefined method factory“. We of course haven’t defined any method called factory

The receiver for the “factory” method

Notice how the error message says undefined method `factory' for main:Object. What’s the main:Object part all about?

Sending a message vs. calling a method

In Ruby you’ll often hear people talk about “sending a message to an object” rather than “calling a method on an object”. The distinction between these things is subtle but significant.

Consider the following bit of code:

a = Array.new(5) # [nil, nil, nil, nil, nil]
a.to_s
a.to_i

The a variable is an instance of the Array class. When we do a.to_s, we’re sending the to_s message to the a object. The a object will happily respond to the to_s message and return a stringified version of the array: "[nil, nil, nil, nil, nil]"

The a object does not respond to (for example) the to_i method. If we send to_i to a, we get an error:

undefined method `to_i' for [nil, nil, nil, nil, nil]:Array

Notice the format of the last part of the error message. It’s value of receiving object:class of receiving object.

Understanding main:Object

In Ruby, every message that gets passed has a receiver, even when it doesn’t seem like there would be. When we do e.g. a.to_s, the receiver is obvious: it’s a. What about when we just call e.g. puts?

When we send a message that doesn’t explicitly have a receiver object, the receiver is a special object called main. That’s why when we call factory we get an error message that says undefined local variable or method `factory' for main:Object. The interpreter sends the factory message to main because we’re not explicitly specifying any other receiver object.

Changing the receiver of the factory message

If we want our program to work, we’re going to have to change the receiver of factory from main to something else. If the receiver were just main, then our factory method would have to just be defined out in the open, not as part of any object. If the factory method is not defined as part of any object, then it can’t easily share any data with any object, and we’ll have a pretty tough time.

We can change the receiver of the factory message by using a Ruby method called instance_exec.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

The instance_exec method will execute our block in the context of self. We can see that now, when we run our file, our error message has changed. The receiver object is no longer main. It’s now MyFactoryBot.

undefined method `factory' for MyFactoryBot:Class (NoMethodError)

Let’s add the factory method to MyFactoryBot.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now we don’t get any errors.

We know we’ll also need to invoke the block inside of the factory call, so let’s use another instance_exec inside of the factory method to do that.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now it’s complaining, unsurprisingly, that there’s no method called first_name.

undefined method `first_name' for MyFactoryBot:Class (NoMethodError)

Adding the first_name and last_name methods

We of course need to define methods called first_name and last_name somewhere. We could conceivably add them on the MyFactoryBot class, but it would probably work out better to have a separate instance for each factory that we define, since of course a real application will have way more than just one factory.

Let’s make it so that the factory method creates an instance of a new class called MyFactory and then invokes the block on MyFactory.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

We of course still haven’t actually defined a method called first_name, so we still get an error about that, although the receiver is not MyFactory rather than MyFactoryBot.

undefined method `first_name' for #<MyFactory:0x0000ffff9a955ae8> (NoMethodError)

Let’s define a first_name and last_name method on MyFactory in order to make this error message go away.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now we get no errors.

Providing values for first_name and last_name

Let’s now come back and uncomment the last two lines of the file, the lines that output values for first_name and last_name.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

We get an error.

undefined method `first_name' for nil:NilClass (NoMethodError)

Remember that every message in Ruby has a receiver. Apparently in this case the receiver for first_name when we do user.first_name is nil. In other words, user is nil. That’s obviously not going to work out. It does make sense, though, because MyFactoryBot.create(:user) has no return value.

Let’s try making it so that MyFactoryBot#create returns our instance of MyFactory.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now there are no errors but, but there are also no values present in the output we see.

First name: 
Last name:

Let’s have a closer look and see exactly what user is.

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

The user object is an instance of MyFactory.

MyFactory
First name: 
Last name:

This was maybe a good intermediate step but we of course want user to be an instance of our Active Record User class, just like how the real Factory Bot does. Let’s change MyFactoryBot#create so that it returns an instance of User.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

This gives no errors but there are still no values present.

In order for the first_name and last_name methods to return values, let’s make it so each one calls the block that it’s given.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name(&block)
    @user.first_name = block.call
  end

  def last_name(&block)
    @user.last_name = block.call
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we see “John” and “Smith” as our first and last name values.

User
First name: John
Last name: Smith

Generalizing the factory

What if we wanted to add this? It wouldn’t work.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
    email { "john.smith@example.com" }
  end
end

Obviously we can’t just have hard-coded first_name and last_name methods in our factory. We need to make it so our factory can respond to any messages that are sent to it (provided of course that those messages correspond to actual attributes on our Active Record models).

Let’s take the first step toward generalizing our methods. Instead of @user.first_name = block.call, we’ll do @user.send("first_name=", block.call), which is equivalent.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name(&block)
    @user.send("first_name=", block.call)
  end

  def last_name(&block)
    @user.send("last_name=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

We can go even further than this. Rather than having methods called first_name and last_name, we can use Ruby’s method_missing to dynamically respond to any message that gets sent.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def method_missing(attr, *args, &block)
    # If the message that's sent is e.g. first_name, then
    # the value of attr will be :first_name, and the value
    # of "#{attr}=" will be "first_name=".
    @user.send("#{attr}=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
    email { "john.smith@example.com" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"
puts "Email: #{user.email}"

If we run our file again, we can see that our method_missing code handles not only first_name and last_name but email as well.

First name: John
Last name: Smith
Email: john.smith@example.com

More generalization

What if we want to have more factories besides just one for User? I happen to have a model in my Rails app called Website. What if I wanted to have a factory for that?

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

MyFactoryBot.define do
  factory :website do
    name { "Google" }
    url { "www.google.com" }
  end
end

Right now, it wouldn’t work because I have the User class hard-coded in my factory.

undefined method `name=' for #<User:0x0000ffff8f950d28> (NoMethodError)

Let’s make it so that rather than hard-coding User, we set the class dynamically. Let’s also make it so that the argument you pass to the factory method (e.g. :user or :website) retries the appropriate factory. We can accomplish this by putting our factories into a hash.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factories ||= {}
    @factories[model_sym] = MyFactory.new(model_sym)
    @factories[model_sym].instance_exec(&block)
  end

  def self.create(model_sym)
    @factories[model_sym].record
  end
end

class MyFactory
  attr_reader :record

  def initialize(model_sym)
    @record = model_sym.to_s.classify.constantize.new
  end

  def method_missing(attr, *args, &block)
    @record.send("#{attr}=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

MyFactoryBot.define do
  factory :website do
    name { "Google" }
    url { "www.google.com" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"
puts

website = MyFactoryBot.create(:website)
puts website.class.name
puts "Name: #{website.name}"
puts "URL: #{website.url}"

Now, when we run our file, both factories work.

First name: John
Last name: Smith
Email: john.smith@example.com

Name: Google
URL: www.google.com

Takeaways

  • Factory Bot syntax (and other DSLs) aren’t magic. They’re just Ruby.
  • Blocks can be a powerful way to make your code more expressive and understandable.
  • In Ruby, it’s often more useful to talk about “sending messages to objects” rather than “calling methods on objects”.
  • Every message sent in Ruby has a receiver. When a receiver is not explicitly specified, the receiver is a special object called main.
  • The method_missing method can allow your objects to respond to messages dynamically.

Testing implementation vs. behavior in Rails

One of the biggest mistakes I see in Rails testing (especially model testing) is testing implementation instead of behavior.

Let’s say I run a factory that makes cars. I want to make sure that each car I make actually works. How should I go about making sure each car works?

Here are two possible methods.

Testing implementation

One way I could try to make sure each car works is to check to see if it has all the right stuff. I could check for the presence of an engine, an ignition, a brake system, wheels, and everything else that’s needed in order to get from point A to point B. If the car has all this stuff, then it’s all set.

Testing behavior

Another way I could try to make sure the car works is to start it up and try to drive it somewhere.

The first way is bad and the second way is good. Here’s why.

Why testing behavior is better

If I “verify” that my car works by checking for the presence of various parts, then I haven’t really actually verified anything. I haven’t demonstrated that the system under test (the car) actually meets spec (can drive).

If I test the car by actually driving it, then the questions of whether the car has various components become moot. If for example the car can travel down the road, we don’t need to ask if the car has wheels. If it didn’t have wheels it wouldn’t be moving.

All of our “implementation” questions can be translated into more meaningful “behavior” questions.

  • Does it have an ignition? -> Can it start up?
  • Does it have an engine and wheels? -> Can it drive?
  • Does it have brakes? -> Can it stop?

Lastly, behavior tests are better than implementation tests because behavior tests are more loosely coupled to the implementation. I ask “Can it start up?” instead of “Does it have an engine?” then I’m free to, for example, change my car factory from a gasoline-powered car factory to an electric car factory without having to change the set of tests that I perform. In other words, behavior tests enable refactoring.

Different granularities of behavior testing

I want to be sure that my argument isn’t misconstrued as a discussion between unit tests and integration tests. The implementation tests being performed weren’t actually verifying the proper behavior of the various components (engine, ignition, etc.) in isolation. The implementation tests were merely verifying the presence of components.

In fact, performing some “unit” tests and some “integration” tests in my car factory would probably be a good idea. It would be smart to make sure each part works individually before I combine them all. I’d just want to make sure my unit and integration tests were testing behavior, not implementation.

Verifying that my car can drive is a test at the highest granularity. At a lower level of granularity, I might want to verify that the engine works, even before the engine is connected to an actual car. To test my engine, I wouldn’t want to check to see whether the engine has six cylinders, each with a piston inside. Instead I would want to give my engine some fuel and actually see if it runs.

So please don’t confuse the implementation vs. behavior question with the unit test vs. integration test question. We sometimes want to perform unit tests and we sometimes want to perform integration tests, but we pretty much always want to test behavior instead of implementation.

How this relates to Rails

I sometimes come across examples of Rails model tests that I consider pointless. These tests verify the presence of validations, associations, callbacks and other implementation details. These tests are pointless because they don’t actually test anything. Having these tests is like checking for the presence of your engine and brakes rather than actually just driving your car.

It’s also pointless to write system specs that e.g. check for the presence of various form fields and buttons, for the same exact reason.

Takeaway

Test behavior, not implementation.

When to use Factory Bot’s traits versus nested factories

When I wrote about nested factories in Factory Bot, I got a couple of questions to the effect of “why wouldn’t you use traits for that?”

In responses to these questions, I’ll lay out my method for deciding when to use traits versus nested factories.

“Is” versus “has”

My method is pretty simple: If the factory I’m considering has something, I use a trait. If the factory is something, I use a nested factory. Let’s look at a concrete example.

“Has” example (trait)

In the following example I want to create a special kind of user record, a user that has a phone number. The user is still conceptually a regular old user. The only difference is that this user happens to have a value for its phone_number attribute.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    trait :with_phone_number do
      phone_number { Faker::PhoneNumber.phone_number }
    end
  end
end

When I want to create a user that has a phone number, I do so by doing FactoryBot.create(:user, :with_phone_number).

“Is” example (nested factory)

In the following example I want to create a special kind of user record, a user that is a physician. Conceptually, a physician user is not just a regular old user. This type of user is different in kind. A physician user has different capabilities from a regular user and is used in different ways from a regular user.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    factory :physician_user do
      role { 'physician' }
    end
  end
end

When I want to create a physician user, I do so by doing FactoryBot.create(:physician_user). Note the contrast between this and FactoryBot.create(:user, :with_phone_number).

Takeaway

When deciding whether to use a trait or a nested factory, consider whether the record has something or if it is something. If it has something, use a trait. If it is something, use a nested factory.

Nested factories in Factory Bot: what they are and how to use them

Sometimes you want to be able to create records that are 95% the same as the “default” but have one or two small differences. Nested factories can be useful for this purpose.

Physician user example

Let’s use the following as an example. Let’s say that in the majority of your application’s tests that involve a User, you want just a regular old user. Maybe you have 30 such tests.

But for a handful of tests you want to use a special kind of user, a user that indicates that the person using the application is a physician.

In this scenario, the technical difference between a “physician user” and a regular user is that a physician user has a role attribute that’s set to 'physician'. Let’s say there are 6 tests that use a physician user, quite a small proportion of the total.

Options for addressing this challenge

You have the following options for addressing this challenge.

First, you could set the role to “physician” individually on each of your 6 physician user tests. Unfortunately this would be bad for all the reasons that duplication is bad. If the nature of your physician user were ever to change, you’d have to know that you’d have to make the change in all 6 of these places. This solution isn’t great.

Alternatively, you could alter the user factory to always have role set to “physician”. Under this scenario, every user would always be a physician user, even when it’s not necessary. Even if this would function properly from a technical perspective, it would be misleading to any maintainers of the test suite. For any test, it’s a good idea to create the minimum amount of setup data and no more. An outside observer has no way of knowing which parts of a test’s setup data are necessary and which aren’t, and so must assume it’s all potentially necessary. We wouldn’t want to send our future maintainers an untruthful message.

A third option is to create an alternate version of the user factory that inherits from the original, and use that version in our physician tests. In this scenario we’re not duplicating our setup steps, and we’re also not modifying our “default” factory. The name for this in Factory Bot is nested factories.

How nested factories work

Consider the following User factory.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }
  end
end

In addition to the above, we can define a nested factory called :physician_user which will have all the same attributes as our regular users, but with the extra attribute of role with the value of 'physician'.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    factory :physician_user do
      role { 'physician' }
    end
  end
end

To create a physician user we would simply invoke the following:

FactoryBot.create(:physician_user)

Unlike root-level factory names which map to Active Record models, nested factory names are fully arbitrary.

Takeaways

  • If you have setup data needs that are common across tests, it’s usually not a good idea to duplicate the setup. Duplication in test code usually has the same negative consequences as duplication in application code.
  • Each test should have as much setup data as is necessary for that test and no more. Otherwise a maintainer will have a hard time telling what’s necessary and what’s not.
  • Factory Bot’s nested factories can help us keep our test code DRY while continuing to adhere to certain good testing principles.

Dissecting Factory Bot’s factory definition syntax

Mysterious syntax

If you’ve used Factory Bot at all, you’ve seen syntax like this:

FactoryBot.define do
  factory :user do
    first_name { 'John' }
    last_name { 'Smith' }
    email { 'john.smith@example.com' }
  end
end

When I was first getting started with Rails and I wasn’t very familiar with Ruby, I would look at these files and understand what’s going on conceptually but I would have no idea what’s going on syntactically.

Why I didn’t understand this code

There were three barriers to my understanding at that early stage of my Ruby experience:

  • Ruby’s optional parentheses feature, although it can help code be very expressive, can also make it hard for beginners to tell what’s what.
  • I didn’t understand Ruby blocks yet.
  • I didn’t know about dynamically-defined methods.

Let’s look at a modified example that will make some of the syntax clearer.

A modified factory definition example

Below is a version of the above factory definition that’s functionally equivalent but with two syntactical changes.

The first change is that I’ve included parentheses for all method calls. The second is that I’ve changed all block usages to do syntax instead of the shorthand {} syntax.

FactoryBot.define() do # define is a method
  factory(:user) do
    first_name do
      'John'
    end

    last_name do
      'Smith'
    end

    email do
      'john.smith@example.com'
    end
  end
end

You can see here that define and factory are each just methods. Each of the two methods takes a block. (If you’re not very comfortable with blocks yet, check out my post on understanding Ruby blocks.)

first_name, last_name and email are also methods that take blocks, speaking loosely. Before we can talk about those we need to talk about methods versus messages.

Methods and messages

When you call a method on an object, it can be said that you’re sending a message to that object. For example, when you call "5".to_i, you’re sending the message to_i to the object "5", which is of course a String.

In the above case, the message to_i also happens to be a method that’s defined on String. This doesn’t need to be the case though. We could send any message at all to String, String just might not necessarily respond to that particular message. That’s why we have the respond_to? method, to see what messages an object responds to.

The messages an object will respond to need not be limited to the methods that are defined on that object. An object author can use the method_missing method to allow an object to respond to any message that’s sent to it, and respond in any way that the object author chooses.

Factory definitions and messages

What’s likely happening with first_name, last_name and email is that Factory Bot is using method_missing to allow arbitrary messages to be sent, and if the message (e.g. the message of first_name) matches an attribute on the model that the factory is for, then Factory Bot uses the block passed with the message to set the value of that attribute.

Takeaways

  • Factory Bot’s factory definitions are made out of methods and blocks.
  • Adding parentheses to any piece of DSL code can often make the code clearer.
  • Ruby objects can be passed arbitrary messages, and objects can be designed to respond to those messages.

Should I be doing test-driven development?

When I see questions from beginners regarding learning testing, sometimes they seem to conflate testing with test-driven development (TDD). People will say “I have such-and-such question about TDD” but really it’s just a question about testing, nothing to do with TDD specifically.

Other people sometimes ask questions about whether TDD is “better” than writing tests after.

In this post I’ll try to clarify what’s TDD and what’s not. I’ll also explain whether I think it makes sense for testing beginners to try to practice TDD.

Testing != TDD

First of all, at the risk of stating the obvious, testing and TDD aren’t the same thing. TDD is a specific kind of testing practice where you write the tests before you write the code that makes the test pass. (If you want to go deeper into TDD, I highly recommend Kent Beck’s Test Driven Development: By Example.)

Learning vs. incorporating

Another mistake beginners sometimes make is to conflate learning testing with incorporating testing as a habitual part of their development workflow. They feel like they need to start adopting testing practices into their workflow from day one, and if they fail to do that, then they’ve failed at learning testing.

I think it’s more productive to separate the jobs of learning testing and applying testing. It’s not like skiing, where you learn it and do it at the same time. It’s more like basketball, where you practice free throws in your driveway and build some skills that way before you try to play a real game in front of an audience. You’ll get farther in the beginning if you separate the practice from the application of what you’ve learned to production tests. When you get comfortable enough, you can take off the training wheels and get all your practice from writing production tests.

TDD is beneficial but optional

TDD is super helpful in certain scenarios but it’s not something you absolutely need to learn when you’re first learning testing. I think it’s completely appropriate to first learn the fundamentals of testing in general, and then start to learn TDD once you’ve developed a decent level of comfort with testing.

I don’t always practice TDD

I’m not an advocate of practicing TDD 100% of the time in Rails, even for experienced testers. The reason is that when I’m building a new feature, I often have little idea what shape that feature will take, and the most realistic way for me to hammer it into shape is to just start building it. Once I’ve built some of the feature, then I’ll start adding tests. So, a portion of the time, I write my tests after writing my application code.

The place where I find TDD most useful is for model code. I practice TDD in my models a high percentage of the time. Once I’ve put the broad strokes of a feature in place, I’ll usually use TDD to work out the fine-grained aspects of it.

Takeaways

  • Testing and test-driven development aren’t the same thing.
  • When you’re first learning testing, it can be helpful to separate learning testing from applying testing.
  • You don’t need to learn TDD when you’re starting out.
  • I don’t always practice TDD or even advocate practicing TDD 100% of the time. I myself practice TDD maybe 60% of the time.

How do I add tests to an existing Rails project?

One of the most common questions asked by developers new to Rails testing is “How do I add tests to an existing Rails project?”

The answer largely depends on your experience level with testing. Here are my answers based on whether you have little testing experience or if you’re already decently comfortable with testing.

If you have little testing experience

If you have little testing experience, I would suggest getting some practice on a fresh Rails app before trying to introduce testing to the existing Rails project you want to add tests to.

Adding tests to an existing project is a distinct skill from writing tests for new projects. Adding tests to an existing project can be difficult even for very experienced testers, for reasons described below.

At the same time, you probably don’t want to wait a year to learn testing before you start enjoying the benefits of testing on your existing Rails app. What I would suggest is to first start a fresh throwaway Rails app for the purpose of learning testing. Then, once you’ve gotten a little experience there, see if you can apply something to your existing Rails app. Then, if things get too hard in the existing app, switch back to the throwaway app so you can strengthen your skills more. Continue switching back and forth until you don’t need to anymore.

If you’re already comfortable with testing

Here’s how I suggest adding tests to an existing Rails project: 1) develop a shared vision with your team, 2) start with what’s easiest, then 3) expand your test coverage.

Develop a shared vision

Going from no tests to decent test coverage is unfortunately not as simple as just deciding one day that from now on we’re going to write tests.

The team maintaining the codebase needs to decide certain things, like what testing tools they’re going to use and what testing approach they’re going to use.

In other words, if the team wants to go from point A to point B, they have to decide exactly where point B is and how they intend to try to get there.

Start with what’s easiest

When adding tests to a codebase that has few or no tests, it might seem logical to start by adding tests where tests would be most valuable. Or it might seem logical to require all new changes to have tests. Unfortunately, both these ideas have problems.

The features in an application that are most valuable are also likely to be among the most non-trivial. This means that tests for these features will probably be relatively hard to write due to the large amount of setup data needed. Code written without testability in mind can also be difficult to test due to entangled dependencies.

Requiring all new changes to have tests also has problems. New changes aren’t usually independent of existing code. They’re usually quite tangled up. This brings us back to the same problem we’d have adding tests to our most important features: the setup and dependencies make adding tests difficult, sometimes prohibitively so.

What I would do instead is start with what’s easiest. I would look for the simplest CRUD interfaces in the app and add some tests there, even if those particular tests didn’t seem to add much value. The idea isn’t to add valuable tests right from the start but to establish a beachhead that can be expanded upon.

Expand

Once you have a handful of tests for trivial features, you can add tests for increasingly complicated features. This will give you a much better shot at ending up with good test coverage than trying to start with the most valuable features or trying to add tests for all new changes.

The mechanical details

If your existing Rails application doesn’t have any testing infrastructure, I would suggest taking a look at my how I set up a Rails application post. (Remember that it’s possible to apply an application template to an existing project.)

As you add tests to your project starting with the most trivial features, I would suggest starting with system specs as opposed to model specs or any other type of specs. The reason is that system specs are often more straightforward to conceive of and understand. If you’d like a formula you can apply to add system specs to almost any CRUD feature, you can find that here.

Then, as you get deeper into adding tests to your application, I would suggest two resources: Working Effectively with Legacy Code by Michael Feathers and my post about using tests as a tool to wrangle legacy projects. You might not consider your project a legacy project, but the techniques will be useful anyway.

What are all the Rails testing tools and how do I use them?

One of the most common questions for Rails developers new to testing is “What are all the Rails testing tools and how do I use them?”

I’ll explain what the major tools are but I want to preface it by saying that the most important thing to learn to be a successful tester is testing principles, not testing tools. If you think of testing like a taco, the tools are the tortilla and the principles are the stuff inside the taco. The tortilla is essential but it’s really only a vehicle.

The following are the tools I use for my testing.

RSpec

RSpec is a test framework. A test framework is what gives us a structure for writing our tests as well as the ability to run our tests.

There are other test frameworks but RSpec is the most popular one for commercial Rails projects. The second most popular test framework is Minitest.

Test frameworks differ syntactically but the testing principles and practices are going to be pretty much the same no matter what framework you’re using. (If you’re not sure whether you should learn RSpec or Minitest, I write about that here.)

Factory Bot

One of the challenges of Rails testing is generating test data. For example, if you’re writing a test that logs a user in and then takes some action, you’re going to have to create a user in the database at the beginning of the test. Many tests require much more complicated test data setup.

There are two common ways of generating test data in Rails tests: fixtures and factories.

Fixtures

Fixtures typically take the form of one or more YAML files with some hard-coded data. The data is translated into database records one time, before any of the tests are run, and then deleted afterward. (This happens in a separate test database instance of course.)

Factories

With factories, database data is generated specifically for each test. Instead of loading all the data once at the beginning and deleting it at the end, data is inserted before each test case and then deleted before the next test case starts. (More precisely, the data isn’t deleted, but rather the test is run inside a database transaction and the data is never committed in the first place, but that’s a mechanical detail that’s not important right now.)

Relative merits of fixtures and factories

I tend to prefer factories because I like having my data generation right inside my test, close to where the test is happening. With fixtures the data setup is too distant from where the test happens.

In my experience, for whatever reason, most people who use RSpec use factories and most people who use Minitest use fixtures. If you’d like to learn more about factories and fixtures, I write more about it here.

Capybara

Some Rails tests only exercise Ruby code. Other tests actually open up a browser and simulate user clicks and keystrokes.

Simulating user input this way requires us to use some sort of tool to manipulate the browser. Capybara is a library that uses Ruby to wrap a driver (usually the Selenium driver), letting us simulate clicks and keystrokes using convenient Ruby methods.

For more examples of how to use Capybara, go here.

VCR and WebMock

One principle of testing is that tests should be deterministic, meaning they run the same way every time no matter what.

When an application’s behavior depends on external services (e.g. a third-party API like Stripe) it makes it harder to have deterministic tests. The tests can be made to fail by an internet connection failure or a temporary outage of the external service.

Tools like VCR and WebMock can help smooth out these challenges. VCR can let us run our tests against the real external service, but capture all the service’s responses in local files so that subsequent test runs don’t talk to the external service but rather just go off of the saved responses. That way, even if the internet connection fails or the service goes down, the tests still work.

WebMock is a tool that serves a similar purpose, although I usually use it in a more limited way. I don’t consider my test suite to be deterministic unless it doesn’t talk to the network at all, so I use WebMock to enforce that my test suite isn’t making any network requests.

Tools I don’t use

Cucumber is a somewhat popular tool when it comes to acceptance testing. It’s my view that Cucumber adds an extra layer of complexity and indirection without adding any value. Here are some details on why I don’t recommend Cucumber.

I also don’t use Shoulda matchers. Shoulda matchers make it easy and convenient to write certain kinds of tests, but the kinds of tests Shoulda helps you write are not a good kind of test to write in the first place. Shoulda helps you write tests that test your code’s implementation rather than its behavior. Here are more details on why I don’t recommend Shoulda.

Takeaways

Rails testing tools take some time to learn, but the important part (and perhaps more difficult part) is learning testing principles.

If you’re just getting started with Rails testing, the next step I would suggest is to learn about the different types of Rails tests and when to use them.

What are the different kinds of Rails tests and when should I use each?

When starting out with Rails testing, it’s hard to know where to start.

First, there’s the decision of which framework to use. Then, if you’ve chosen RSpec (which most people do), you’re presented with a bewildering set of possible test types to use.

In this post I’ll show you what types of tests there are. I’ll show you which ones you should use and which ones you can ignore. Since most commercial Rails projects use RSpec, I’m going to focus on the eight types of tests that the RSpec library offers. (Although if I were to use Minitest, my strategy regarding test types would be pretty much the same.)

The eight types of RSpec specs

The RSpec library offers a lot of different spec types.

  • Model specs
  • System specs/feature specs*
  • Request specs/controller specs*
  • Helper specs
  • View specs
  • Routing specs
  • Mailer specs
  • Job specs

There are two lines with asterisks. These are cases where the RSpec team decreed one spec type obsolete and replaced it with a new type. I’m only including those ones for completeness.

So the up-to-date list is really the following.

  • Model specs
  • System specs
  • Request specs
  • Helper specs
  • View specs
  • Routing specs
  • Mailer specs
  • Job specs

Here’s when I use each.

  • Model specs – always
  • System specs – always
  • Request specs – rarely
  • Helper specs – rarely
  • View specs – never
  • Routing specs – never
  • Mailer specs – never
  • Job specs – never

Let’s talk about each of these spec types in detail. I’ll explain why I use the ones I use and why I ignore the ones I ignore.

Spec types I always use

Believe it or not, the overwhelming majority of the Rails tests I write make use of just two of the eight different spec types offered by RSpec. You might think that this would leave large gaps in my test coverage but it doesn’t. My test coverage is consistently above 95%.

System specs

System specs are “high-level” tests that simulate a user’s keystrokes and mouse clicks. System specs literally open up a browser window (although perhaps an invisible browser window if the tests are run “headlessly”) and use certain tools to manipulate the browser to exercise your application through simulated user input.

The reason I find system specs so valuable is that they test my whole stack, not just a slice of it, and they test my application in the same exact way that a real user will be using it. System specs are the only type of test that give me confidence my whole application really works.

I write so many system specs that I’ve developed a repeatable formula for adding system specs to any new CRUD feature.

Model specs

Even though system specs are indispensable, they’re not without drawbacks. System specs are somewhat “heavy”. They’re often more work to write and more expensive to run than other types of tests. For this reason I like to cover my features with a small number of coarse-grained system specs and a comparatively large number of fine-grained model specs.

As the name implies, model specs are for testing models. I tend to only bring model specs into the picture once a model has reached a certain level of “maturity”. At the beginning of a model’s life, it might have all its needs covered by built-in Rails functionality and not need any methods of its own. Some people write tests for things like associations and validations but I don’t because I find those types of tests to be pointless.

I use model specs to test my models’ methods. When I do so, I tend to use a test-first approach and write a failing test before I add a new line of code so that I’m sure every bit of code in my model is covered by a test.

Spec types I rarely use

Request specs

Request specs are more or less a way to test controller actions in isolation. I tend not to use request specs much because in most cases they would be redundant to my system specs. If I have system specs covering all my features, then of course a broken controller would fail one or more of my tests, making tests specifically for my controllers unnecessary.

I also try to keep my controllers sufficiently simple as to not call for tests of their own.

There are just three scenarios in which I do use request specs. First: If I’m working on a legacy project with fat controllers, sometimes I’ll use request specs to help me harness and refactor all that controller code. Second: If I’m working on an API-only Rails app, then system specs are physically impossible and I drop down to request specs instead. Lastly, if it’s just too awkward or expensive to use a system spec in a certain case then I’ll use a request spec instead. I write more about my reasoning here.

Helper specs

The reason I rarely write helper specs is simple: I rarely write helpers.

Spec types I never use

View specs and routing specs

I find view specs and routing specs to be redundant to system specs. If something is wrong with one of my views or routes, it’s highly likely that one of my system specs will catch the problem.

Mailer specs and job specs

I don’t write mailer specs or job specs because I try very hard to make all my mailers and background jobs one-liners (or close). I don’t think mailers and background jobs should do things, I think they should only call things. This is because mailers and background jobs are mechanical devices, not code organization devices.

To test my mailers and background jobs, I put their code into a PORO model and write tests for that PORO.

Takeaways

RSpec offers a lot of different spec types but you can typically meet 98% of your needs with just system specs and model specs.

If you’re a total beginner, I’d suggest starting with system specs.