Category Archives: Automated Testing

The four phases of a test

When writing tests, or reading other people’s tests, it can be helpful to understand that tests are often structured in four distinct phases.

These phases are:

  1. Setup
  2. Exercise
  3. Assertion
  4. Teardown

Let’s illustrate these four phases using an example.

Test phase example

Let’s say we have an application that has a list of users that can receive messages. Only active users are allowed to receive messages. So, we need to assert that when a user is inactive, that user can’t receive messages.

Here’s how this test might go:

  1. Create a User record (setup)
  2. Set the user’s “active” status to false (exercise)
  3. Assert that the user is not “messageable” (assertion)
  4. Delete the User record we created in step 1 (teardown)

In parallel with this example, I’ll also use another example which is somewhat silly but also less abstract. Let’s imagine we’re designing a sharp-shooting robot that can fire a bow and accurately hit a target with an arrow. In order to test our robot’s design, we might:

  1. Get a fresh prototype of the robot from the machine shop (setup)
  2. Allow the robot to fire an arrow (exercise)
  3. Look at the target to make sure it was hit by the arrow (assertion)
  4. Return the prototype to the machine shop for disassembly (teardown)

Now let’s take a look at each step in more detail.

The purpose of each test phase

Setup

The setup phase typically creates all the data that’s needed in order for the test to operate. (There are other things that could conceivably happen during a setup phase but for our current purposes we can think of the setup phase’s role as being to put data in place.)In our case, the creation of the User record is all that’s involved in the setup step, although more complicated tests could of course create any number of database records and potentially establish relationships among them.

Exercise

The exercise phase walks through the motions of the feature we want to test. With our robot example, the exercise phase is when the robot fires the arrow. With our messaging example, the exercise phase is when the user gets put in an inactive state.

Side note: the distinction between setup and exercise may seem blurry, and indeed it sometimes is, especially in low-level tests like our current example. If someone were to argue that setting the user to inactive should actually be part of the setup, I’m not sure how I’d refute them. To help with the distinction in this case, imagine if we instead were writing an integration test that actually opened up a browser and simulated clicks. For this test, our setup would be the same (create a user record) but our exercise might be different. We might visit a settings page, uncheck an “active” checkbox, then save the form.

Assertion

The assertion phase is basically what all the other phases exist in support of. The assertion is the actual test part of the test, the thing that determines whether the test passes or fails.

Teardown

Each test needs to clean up after itself. If it didn’t, then each test would potentially pollute the world in which the test is running and affect the outcome of later tests, making the tests non-deterministic. We don’t want this. We want deterministic tests, i.e. tests that behave the same exact way every single time no matter what. The only thing that should make a test go from passing to failing or vice-versa is if the behavior that the test tests changes.

In reality, Rails tests tend not to have an explicit teardown step. The main pollutant we have to worry about with our tests is database data that gets left behind. RSpec is capable of taking care of this problem for us by running each test in a database transaction. The transaction starts before each test is run and aborts after the test finishes. So really, the data never gets permanently persisted in the first place. So although I’m mentioning the teardown step here for completeness’ sake, you’re unlikely to see it in the wild.

A concrete example

See if you can identify the phases in the following RSpec test.

RSpec.describe User do
  let!(:user) { User.create!(email: 'test@example.com') }

  describe '#messageable?' do
    context 'is inactive' do
      it 'is false' do
        user.update!(active: false)
        expect(user.messageable?).to be false
        user.destroy!
      end
    end
  end
end

Here’s my annotated version.

RSpec.describe User do
  let!(:user) { User.create!(email: 'test@example.com') } # setup

  describe '#messageable?' do
    context 'is inactive' do
      it 'is false' do
        user.update!(active: false)           # exercise
        expect(user.messageable?).to be false # assertion
        user.destroy!                         # teardown
      end
    end
  end
end

Takeaway

Being familiar with the four phases of a test can help you overcome the writer’s block that testers sometimes feel when staring at a blank editor. “Write the setup” is an easier job than “write the whole test”.

Understanding the four phases of a test can also help make it easier to parse the meaning of existing tests.

When I do TDD and when I don’t

Some developers advocate doing test-driven development 100% of the time. Other developers think TDD is for the birds and don’t do it at all. Still other developers go in the middle and practice TDD more than 0% of the time but less than 100% of the time.

I personally am in the camp of practicing TDD some of the time but not all. Here’s my reasoning.

When TDD makes sense to me

It’s not the case that I use TDD, or even write tests at all, for every single project I work on. But I do pretty much always program in feedback loops.

Feedback loops

The “feedback loop method” is where work as follows. First, I think of a tiny goal that I want to accomplish (e.g. make “hello world” appear on the screen). Then I decide on a manual test I can perform in order to see if that goal is accomplished (e.g. refresh the page and observe). Then I perform the test, write some code to try to make the test pass, perform the test again, and repeat the process with a new goal.

TDD == automated feedback loops

The way I view TDD is that it’s just the automated version of the manual work I was going to do anyway. Instead of making a to-do note that says “make ‘hello world’ appear on the screen” and then manually refreshing the page to see if it’s there, I write a test that expects “hello world” to appear on the screen. All the other steps are the exact same.

I’ve found that TDD works great for me when I’m working on what you might call “crisply-defined” work. In other words, the requirements I’m working to fulfill are known and specified. I find that I’ve found that there are other scenarios where TDD doesn’t work so great for me.

When TDD doesn’t make sense to me

Coding as production vs. coding as thinking

It’s easy to think that the reason to write code is to create a work product. But that’s certainly not the only reason to write code. Code isn’t just a medium for producing a product. It’s also a medium for thinking.

This is the idea behind a “spike” in Agile programming. When you’re doing a spike, you have no necessary intention to actually keep any of the code you’re writing. You’re just exploring. You’re seeing what it looks like when you do this or how it feels when you do that.

You can think of coding kind of like playing a piano. Sometimes you have some pieces of music already in your head and you’re trying to record an album. Other times you’re just messing around to see you can come up with any music worth recording. These are two very different modes of engaging with your instrument. Both are very necessary in order to ultimately record some music.

TDD doesn’t mix great with spikes

I often find that a spike phase is necessary when I’m coding, for example, a feature with known big-picture requirements but unknown UI specifics. In that case my test would be so full of guesses and placeholders that it would be kind of a joke of a test, and it wouldn’t help me much. In these cases I give myself permission to forego the testing during the spike period. I come back after I have some working code and backfill the tests.

Takeaways

  • I don’t practice TDD 100% of the time. (I believe I do practice TDD the vast majority of the time though.)
  • I view TDD as the automated version of the coding workflow that I already use anyway.
  • Producing a work product is not the only reason to write code. Code can also be a medium for thinking.
  • When I’m in the mode of using coding as a way to think, I find that the benefits of TDD don’t really apply.

If you want to learn testing, first learn how to program in feedback loops

Most beginner programmers (and even many experienced programmers) take a slow, painful, wasteful approach to programming.

The wasteful way

The way many programmers code is to spend a bunch of time writing code without checking to see that it works, then finally run the program once they’ve accumulated many lines of code. The program inevitably fails.

Next, the programmer sits and puzzles over what might have gone wrong. Since the programmer wrote a lot of code without checking it, there’s a lot of stuff that could possibly be the culprit, and therefore the debugging process is slow and painful.

The debugging process is usually not a systematic one but rather a guessing game. “Maybe it’s this. Nope it’s not that. Maybe it’s this other thing. Nope, it’s not that either. Hmm.” As the clock ticks, frustration mounts, and maybe a little desperation sets in. It’s not fast and it’s not fun.

The smarter way: feedback loops

Instead of working in the slow, painful, wasteful way described above, you can work in feedback loops. As I described in my other post about feedback loops, the feedback loop process goes like this:

  1. Decide what you want to accomplish
  2. Devise a manual test you can perform to see if #1 is done
  3. Perform the test from step 2
  4. Write a line of code
  5. Repeat test from step 2 until it passes
  6. Repeat from step 1 with a new goal

When you use the feedback loop method, it’s hard to run too far astray. If you only write a little bit of code at a time and you keep everything working at all times, then you’re guaranteed to always have a program that’s either fully working or very close to fully working.

Feedback loops and automated testing

Automated testing is just the practice of coding using feedback loops, but with the testing step automated.

Here’s how the feedback loop would go with automated tests involved. The automated test parts are included in bold.

  1. Decide what you want to accomplish
  2. Devise a manual test you can perform to see if #1 is done (write a test)
  3. Perform the test from step 2 (run the test)
  4. Write a line of code
  5. Repeat test from step 2 until it passes (run the test again)
  6. Repeat from step 1 with a new goal

Obviously there’s also a lot of technical knowledge that’s needed in order to write automated tests. For example, there are test frameworks that enable automated testing, there are libraries that help your tests interact with a browser, and there are libraries that help with generating test data. But more important than any particular tool are the principles behind automated testing.

Perhaps the most important idea behind automated testing is the feedback loop. And luckily for you if you’re a beginner, you can learn how to program in feedback loops without having to learn anything to do with automated testing yet. And once you do, writing automated tests will feel much more natural.

What should I NOT write tests for?

One of the most common questions about testing, including what to write tests for, is what NOT to write tests for.

When people ask me what to write tests for, my honest but maybe not very helpful answer is “basically everything”. But I don’t test literally absolutely everything. There are some cases when I choose to skip tests.

My criteria for skipping a test

My habit of writing tests for everything leads me to write tests for my code by default. For most of the code I write, I actually finder it harder to write the code without tests than with tests.

But sometimes I get lazy or my instincts tell me that a test would be a waste of time. When I’m feeling like this, I ask myself three questions:

  1. If this code were to misbehave, would it fail silently?
  2. If this code were to misbehave, how bad would the consequences be?
  3. If this code were to misbehave, how frequently would it fail?
  4. How costly would it be to write a test?

If the code I’m working on would fail in a very obvious way, and the consequences are minor, and the failure would only happen once a year, and the test would be very costly to write, then that’s a case where I would probably skip the test.

On the other hand, if the code would fail silently, OR if the consequences would be bad, OR if the failure would be frequent, OR if it’s super easy to write the test, then I would just write the test.

You could think of these questions as a boolean expression where all the items get OR’d together:

“Should I write a test?” formula (boolean OR)

  1. Might fail silently?
  2. Consequences might be bad?
  3. Might fail frequently?
  4. Test is easy to write?

If any one of the items is true, then the whole boolean expression is true and I go ahead and write the test. Otherwise I skip the test without guilt or worry.

Behavior vs. implementation

There are also entire types of tests I avoid.

I don’t test for things like the presence of associations, the presence of methods, etc. Such tests are pointless. Rather, I test the behavior that these things enable. Testing the behavior is the only way you can really be sure anything works.

Takeaways

  • I’ll skip a test if and only if the feature won’t fail silently, the consequences won’t be bad, the failure won’t occur frequently, and the test is expensive to write.
  • I don’t test implementation details, but rather I test the behaviors that the implementations enable.

Understanding Factory Bot syntax by coding your own Factory Bot

What we’re going to do and why

When you look at a Factory Bot factory definition, the syntax might look somewhat mysterious. Here’s an example of such a factory.

FactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

The goal of this tutorial is to demystify this syntax. The way we’ll do this is to write our own implementation of Factory Bot from scratch. Or, more precisely, we’ll write an implementation of something that behaves indistinguishably from Factory Bot for a few narrow use cases.

Concepts we’ll learn about

Blocks

Factory Bot syntax makes heavy use of blocks. In this post we’ll learn a little bit about how blocks work.

Message sending

We’ll learn the nuanced distinction between calling a method on an object and sending a message to an object.

method_missing

We’ll learn how to use Ruby’s method_missing feature so that we can define methods for objects dynamically.

Our objective

Our goal with this post will be to write some code that makes the factory below actually work. Notice that this factory is indistinguishable from a Factory Bot factory except for the fact that it starts with MyFactoryBot rather than FactoryBot.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

In addition to this factory code, we’ll also need some code that exercises the factory.

Exercising the factory

Here’s some code that will exercise our factory to make sure it actually works. Just like how our factory definition mirrors an actual Factory Bot factory definition, the below code mirrors how we would use a Factory Bot factory.

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

After we get our factory working properly, the above code should produce the following output.

First name: John
Last name: Smith

Let’s get started.

How to follow along

If you’d like to code along with this tutorial, I have a sample project that you can use.

The code in this tutorial depends on a certain Rails project, so I’ve created a GitHub repo at https://github.com/jasonswett/my_factory_bot where there’s a Dockerized Rails app, the same exact Rails app that I used to write this tutorial. You’ll find instructions to set up the project on the GitHub page.

The approach

We’ll be writing our “Factory Bot clone” code using a (silly) development methodology that I call “error-driven development”. Error-driven development works like this: you write a piece of code and try to run it. If you get an error when you run the code, you write just enough code to fix that particular error, and nothing more. You repeat this process until you have the result you want.

The reason I sometimes like to code this way is that it prevents me from writing any code that hasn’t been (manually) tested. Surprisingly enough, this “methodology” actually works pretty well.

Building the factory

The first thing to do is to create a file called my_factory_bot.rb and put it at the Rails project root.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Then we’ll run the code like this:

$ rails run my_factory_bot.rb

The first thing we’ll see is an error saying that MyFactoryBot is not defined.

uninitialized constant MyFactoryBot (NameError)

This is of course true. We haven’t yet defined something called MyFactoryBot. So, in the spirit of practicing “error-driven development”, let’s write enough code to make this particular error go away and nothing more.

class MyFactoryBot
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now, if we run the code again (using the same rails run command from above), we get a different error.

undefined method `define' for MyFactoryBot:Class (NoMethodError)

This is also true. The MyFactoryBot class doesn’t have a method called define. So let’s define it.

class MyFactoryBot
  def self.define
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we get a new error.

undefined method `create' for MyFactoryBot:Class (NoMethodError)

This of course comes from the user = MyFactoryBot.create(:user) line. Let’s define a create method in order to make this error go away. Since we’re passing in an argument, :user, when we call create, we’ll need to specify a parameter for the create method. I’m calling the parameter model_sym since it’s a symbol that corresponds to the model that the factory is targeting.

class MyFactoryBot
  def self.define
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we get an error for the next line.

undefined method `first_name' for nil:NilClass (NoMethodError)

We’ll deal with this error, but not just yet, because this one will be a little bit tricky, and there are some certain other things that will make sense to do first. Let’s temporarily comment out the lines that call first_name and last_name.

class MyFactoryBot
  def self.define
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now, if we run the file again, we get no errors.

Making it so the factory block gets called

Right now, the block inside of MyFactoryBot.define isn’t getting used at all. Let’s add block.call to the defined method so that the block gets called.

class MyFactoryBot
  def self.define(&block)
    block.call
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now when we run the file we get the following error.

undefined method `factory' for main:Object (NoMethodError)

It makes sense that we would get an error that says “undefined method factory“. We of course haven’t defined any method called factory

The receiver for the “factory” method

Notice how the error message says undefined method `factory' for main:Object. What’s the main:Object part all about?

Sending a message vs. calling a method

In Ruby you’ll often hear people talk about “sending a message to an object” rather than “calling a method on an object”. The distinction between these things is subtle but significant.

Consider the following bit of code:

a = Array.new(5) # [nil, nil, nil, nil, nil]
a.to_s
a.to_i

The a variable is an instance of the Array class. When we do a.to_s, we’re sending the to_s message to the a object. The a object will happily respond to the to_s message and return a stringified version of the array: "[nil, nil, nil, nil, nil]"

The a object does not respond to (for example) the to_i method. If we send to_i to a, we get an error:

undefined method `to_i' for [nil, nil, nil, nil, nil]:Array

Notice the format of the last part of the error message. It’s value of receiving object:class of receiving object.

Understanding main:Object

In Ruby, every message that gets passed has a receiver, even when it doesn’t seem like there would be. When we do e.g. a.to_s, the receiver is obvious: it’s a. What about when we just call e.g. puts?

When we send a message that doesn’t explicitly have a receiver object, the receiver is a special object called main. That’s why when we call factory we get an error message that says undefined local variable or method `factory' for main:Object. The interpreter sends the factory message to main because we’re not explicitly specifying any other receiver object.

Changing the receiver of the factory message

If we want our program to work, we’re going to have to change the receiver of factory from main to something else. If the receiver were just main, then our factory method would have to just be defined out in the open, not as part of any object. If the factory method is not defined as part of any object, then it can’t easily share any data with any object, and we’ll have a pretty tough time.

We can change the receiver of the factory message by using a Ruby method called instance_exec.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

The instance_exec method will execute our block in the context of self. We can see that now, when we run our file, our error message has changed. The receiver object is no longer main. It’s now MyFactoryBot.

undefined method `factory' for MyFactoryBot:Class (NoMethodError)

Let’s add the factory method to MyFactoryBot.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now we don’t get any errors.

We know we’ll also need to invoke the block inside of the factory call, so let’s use another instance_exec inside of the factory method to do that.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now it’s complaining, unsurprisingly, that there’s no method called first_name.

undefined method `first_name' for MyFactoryBot:Class (NoMethodError)

Adding the first_name and last_name methods

We of course need to define methods called first_name and last_name somewhere. We could conceivably add them on the MyFactoryBot class, but it would probably work out better to have a separate instance for each factory that we define, since of course a real application will have way more than just one factory.

Let’s make it so that the factory method creates an instance of a new class called MyFactory and then invokes the block on MyFactory.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

We of course still haven’t actually defined a method called first_name, so we still get an error about that, although the receiver is not MyFactory rather than MyFactoryBot.

undefined method `first_name' for #<MyFactory:0x0000ffff9a955ae8> (NoMethodError)

Let’s define a first_name and last_name method on MyFactory in order to make this error message go away.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
#puts "First name: #{user.first_name}"
#puts "Last name: #{user.last_name}"

Now we get no errors.

Providing values for first_name and last_name

Let’s now come back and uncomment the last two lines of the file, the lines that output values for first_name and last_name.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    factory = MyFactory.new
    factory.instance_exec(&block)
  end

  def self.create(model_sym)
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

We get an error.

undefined method `first_name' for nil:NilClass (NoMethodError)

Remember that every message in Ruby has a receiver. Apparently in this case the receiver for first_name when we do user.first_name is nil. In other words, user is nil. That’s obviously not going to work out. It does make sense, though, because MyFactoryBot.create(:user) has no return value.

Let’s try making it so that MyFactoryBot#create returns our instance of MyFactory.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory
  end
end

class MyFactory
  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now there are no errors but, but there are also no values present in the output we see.

First name: 
Last name:

Let’s have a closer look and see exactly what user is.

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

The user object is an instance of MyFactory.

MyFactory
First name: 
Last name:

This was maybe a good intermediate step but we of course want user to be an instance of our Active Record User class, just like how the real Factory Bot does. Let’s change MyFactoryBot#create so that it returns an instance of User.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name
  end

  def last_name
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

This gives no errors but there are still no values present.

In order for the first_name and last_name methods to return values, let’s make it so each one calls the block that it’s given.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name(&block)
    @user.first_name = block.call
  end

  def last_name(&block)
    @user.last_name = block.call
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

Now we see “John” and “Smith” as our first and last name values.

User
First name: John
Last name: Smith

Generalizing the factory

What if we wanted to add this? It wouldn’t work.

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
    email { "john.smith@example.com" }
  end
end

Obviously we can’t just have hard-coded first_name and last_name methods in our factory. We need to make it so our factory can respond to any messages that are sent to it (provided of course that those messages correspond to actual attributes on our Active Record models).

Let’s take the first step toward generalizing our methods. Instead of @user.first_name = block.call, we’ll do @user.send("first_name=", block.call), which is equivalent.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def first_name(&block)
    @user.send("first_name=", block.call)
  end

  def last_name(&block)
    @user.send("last_name=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"

We can go even further than this. Rather than having methods called first_name and last_name, we can use Ruby’s method_missing to dynamically respond to any message that gets sent.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factory = MyFactory.new
    @factory.instance_exec(&block)
  end

  def self.create(model_sym)
    @factory.user
  end
end

class MyFactory
  attr_reader :user

  def initialize
    @user = User.new
  end

  def method_missing(attr, *args, &block)
    # If the message that's sent is e.g. first_name, then
    # the value of attr will be :first_name, and the value
    # of "#{attr}=" will be "first_name=".
    @user.send("#{attr}=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
    email { "john.smith@example.com" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"
puts "Email: #{user.email}"

If we run our file again, we can see that our method_missing code handles not only first_name and last_name but email as well.

First name: John
Last name: Smith
Email: john.smith@example.com

More generalization

What if we want to have more factories besides just one for User? I happen to have a model in my Rails app called Website. What if I wanted to have a factory for that?

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

MyFactoryBot.define do
  factory :website do
    name { "Google" }
    url { "www.google.com" }
  end
end

Right now, it wouldn’t work because I have the User class hard-coded in my factory.

undefined method `name=' for #<User:0x0000ffff8f950d28> (NoMethodError)

Let’s make it so that rather than hard-coding User, we set the class dynamically. Let’s also make it so that the argument you pass to the factory method (e.g. :user or :website) retries the appropriate factory. We can accomplish this by putting our factories into a hash.

class MyFactoryBot
  def self.define(&block)
    instance_exec(&block)
  end

  def self.factory(model_sym, &block)
    @factories ||= {}
    @factories[model_sym] = MyFactory.new(model_sym)
    @factories[model_sym].instance_exec(&block)
  end

  def self.create(model_sym)
    @factories[model_sym].record
  end
end

class MyFactory
  attr_reader :record

  def initialize(model_sym)
    @record = model_sym.to_s.classify.constantize.new
  end

  def method_missing(attr, *args, &block)
    @record.send("#{attr}=", block.call)
  end
end

MyFactoryBot.define do
  factory :user do
    first_name { "John" }
    last_name { "Smith" }
  end
end

MyFactoryBot.define do
  factory :website do
    name { "Google" }
    url { "www.google.com" }
  end
end

user = MyFactoryBot.create(:user)
puts user.class.name
puts "First name: #{user.first_name}"
puts "Last name: #{user.last_name}"
puts

website = MyFactoryBot.create(:website)
puts website.class.name
puts "Name: #{website.name}"
puts "URL: #{website.url}"

Now, when we run our file, both factories work.

First name: John
Last name: Smith
Email: john.smith@example.com

Name: Google
URL: www.google.com

Takeaways

  • Factory Bot syntax (and other DSLs) aren’t magic. They’re just Ruby.
  • Blocks can be a powerful way to make your code more expressive and understandable.
  • In Ruby, it’s often more useful to talk about “sending messages to objects” rather than “calling methods on objects”.
  • Every message sent in Ruby has a receiver. When a receiver is not explicitly specified, the receiver is a special object called main.
  • The method_missing method can allow your objects to respond to messages dynamically.

Testing implementation vs. behavior in Rails

One of the biggest mistakes I see in Rails testing (especially model testing) is testing implementation instead of behavior.

Let’s say I run a factory that makes cars. I want to make sure that each car I make actually works. How should I go about making sure each car works?

Here are two possible methods.

Testing implementation

One way I could try to make sure each car works is to check to see if it has all the right stuff. I could check for the presence of an engine, an ignition, a brake system, wheels, and everything else that’s needed in order to get from point A to point B. If the car has all this stuff, then it’s all set.

Testing behavior

Another way I could try to make sure the car works is to start it up and try to drive it somewhere.

The first way is bad and the second way is good. Here’s why.

Why testing behavior is better

If I “verify” that my car works by checking for the presence of various parts, then I haven’t really actually verified anything. I haven’t demonstrated that the system under test (the car) actually meets spec (can drive).

If I test the car by actually driving it, then the questions of whether the car has various components become moot. If for example the car can travel down the road, we don’t need to ask if the car has wheels. If it didn’t have wheels it wouldn’t be moving.

All of our “implementation” questions can be translated into more meaningful “behavior” questions.

  • Does it have an ignition? -> Can it start up?
  • Does it have an engine and wheels? -> Can it drive?
  • Does it have brakes? -> Can it stop?

Lastly, behavior tests are better than implementation tests because behavior tests are more loosely coupled to the implementation. I ask “Can it start up?” instead of “Does it have an engine?” then I’m free to, for example, change my car factory from a gasoline-powered car factory to an electric car factory without having to change the set of tests that I perform. In other words, behavior tests enable refactoring.

Different granularities of behavior testing

I want to be sure that my argument isn’t misconstrued as a discussion between unit tests and integration tests. The implementation tests being performed weren’t actually verifying the proper behavior of the various components (engine, ignition, etc.) in isolation. The implementation tests were merely verifying the presence of components.

In fact, performing some “unit” tests and some “integration” tests in my car factory would probably be a good idea. It would be smart to make sure each part works individually before I combine them all. I’d just want to make sure my unit and integration tests were testing behavior, not implementation.

Verifying that my car can drive is a test at the highest granularity. At a lower level of granularity, I might want to verify that the engine works, even before the engine is connected to an actual car. To test my engine, I wouldn’t want to check to see whether the engine has six cylinders, each with a piston inside. Instead I would want to give my engine some fuel and actually see if it runs.

So please don’t confuse the implementation vs. behavior question with the unit test vs. integration test question. We sometimes want to perform unit tests and we sometimes want to perform integration tests, but we pretty much always want to test behavior instead of implementation.

How this relates to Rails

I sometimes come across examples of Rails model tests that I consider pointless. These tests verify the presence of validations, associations, callbacks and other implementation details. These tests are pointless because they don’t actually test anything. Having these tests is like checking for the presence of your engine and brakes rather than actually just driving your car.

It’s also pointless to write system specs that e.g. check for the presence of various form fields and buttons, for the same exact reason.

Takeaway

Test behavior, not implementation.

When to use Factory Bot’s traits versus nested factories

When I wrote about nested factories in Factory Bot, I got a couple of questions to the effect of “why wouldn’t you use traits for that?”

In responses to these questions, I’ll lay out my method for deciding when to use traits versus nested factories.

“Is” versus “has”

My method is pretty simple: If the factory I’m considering has something, I use a trait. If the factory is something, I use a nested factory. Let’s look at a concrete example.

“Has” example (trait)

In the following example I want to create a special kind of user record, a user that has a phone number. The user is still conceptually a regular old user. The only difference is that this user happens to have a value for its phone_number attribute.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    trait :with_phone_number do
      phone_number { Faker::PhoneNumber.phone_number }
    end
  end
end

When I want to create a user that has a phone number, I do so by doing FactoryBot.create(:user, :with_phone_number).

“Is” example (nested factory)

In the following example I want to create a special kind of user record, a user that is a physician. Conceptually, a physician user is not just a regular old user. This type of user is different in kind. A physician user has different capabilities from a regular user and is used in different ways from a regular user.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    factory :physician_user do
      role { 'physician' }
    end
  end
end

When I want to create a physician user, I do so by doing FactoryBot.create(:physician_user). Note the contrast between this and FactoryBot.create(:user, :with_phone_number).

Takeaway

When deciding whether to use a trait or a nested factory, consider whether the record has something or if it is something. If it has something, use a trait. If it is something, use a nested factory.

Nested factories in Factory Bot: what they are and how to use them

Sometimes you want to be able to create records that are 95% the same as the “default” but have one or two small differences. Nested factories can be useful for this purpose.

Physician user example

Let’s use the following as an example. Let’s say that in the majority of your application’s tests that involve a User, you want just a regular old user. Maybe you have 30 such tests.

But for a handful of tests you want to use a special kind of user, a user that indicates that the person using the application is a physician.

In this scenario, the technical difference between a “physician user” and a regular user is that a physician user has a role attribute that’s set to 'physician'. Let’s say there are 6 tests that use a physician user, quite a small proportion of the total.

Options for addressing this challenge

You have the following options for addressing this challenge.

First, you could set the role to “physician” individually on each of your 6 physician user tests. Unfortunately this would be bad for all the reasons that duplication is bad. If the nature of your physician user were ever to change, you’d have to know that you’d have to make the change in all 6 of these places. This solution isn’t great.

Alternatively, you could alter the user factory to always have role set to “physician”. Under this scenario, every user would always be a physician user, even when it’s not necessary. Even if this would function properly from a technical perspective, it would be misleading to any maintainers of the test suite. For any test, it’s a good idea to create the minimum amount of setup data and no more. An outside observer has no way of knowing which parts of a test’s setup data are necessary and which aren’t, and so must assume it’s all potentially necessary. We wouldn’t want to send our future maintainers an untruthful message.

A third option is to create an alternate version of the user factory that inherits from the original, and use that version in our physician tests. In this scenario we’re not duplicating our setup steps, and we’re also not modifying our “default” factory. The name for this in Factory Bot is nested factories.

How nested factories work

Consider the following User factory.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }
  end
end

In addition to the above, we can define a nested factory called :physician_user which will have all the same attributes as our regular users, but with the extra attribute of role with the value of 'physician'.

FactoryBot.define do
  factory :user do
    username { Faker::Internet.username }
    password { Faker::Internet.password }

    factory :physician_user do
      role { 'physician' }
    end
  end
end

To create a physician user we would simply invoke the following:

FactoryBot.create(:physician_user)

Unlike root-level factory names which map to Active Record models, nested factory names are fully arbitrary.

Takeaways

  • If you have setup data needs that are common across tests, it’s usually not a good idea to duplicate the setup. Duplication in test code usually has the same negative consequences as duplication in application code.
  • Each test should have as much setup data as is necessary for that test and no more. Otherwise a maintainer will have a hard time telling what’s necessary and what’s not.
  • Factory Bot’s nested factories can help us keep our test code DRY while continuing to adhere to certain good testing principles.

Dissecting Factory Bot’s factory definition syntax

Mysterious syntax

If you’ve used Factory Bot at all, you’ve seen syntax like this:

FactoryBot.define do
  factory :user do
    first_name { 'John' }
    last_name { 'Smith' }
    email { 'john.smith@example.com' }
  end
end

When I was first getting started with Rails and I wasn’t very familiar with Ruby, I would look at these files and understand what’s going on conceptually but I would have no idea what’s going on syntactically.

Why I didn’t understand this code

There were three barriers to my understanding at that early stage of my Ruby experience:

  • Ruby’s optional parentheses feature, although it can help code be very expressive, can also make it hard for beginners to tell what’s what.
  • I didn’t understand Ruby blocks yet.
  • I didn’t know about dynamically-defined methods.

Let’s look at a modified example that will make some of the syntax clearer.

A modified factory definition example

Below is a version of the above factory definition that’s functionally equivalent but with two syntactical changes.

The first change is that I’ve included parentheses for all method calls. The second is that I’ve changed all block usages to do syntax instead of the shorthand {} syntax.

FactoryBot.define() do # define is a method
  factory(:user) do
    first_name do
      'John'
    end

    last_name do
      'Smith'
    end

    email do
      'john.smith@example.com'
    end
  end
end

You can see here that define and factory are each just methods. Each of the two methods takes a block. (If you’re not very comfortable with blocks yet, check out my post on understanding Ruby blocks.)

first_name, last_name and email are also methods that take blocks, speaking loosely. Before we can talk about those we need to talk about methods versus messages.

Methods and messages

When you call a method on an object, it can be said that you’re sending a message to that object. For example, when you call "5".to_i, you’re sending the message to_i to the object "5", which is of course a String.

In the above case, the message to_i also happens to be a method that’s defined on String. This doesn’t need to be the case though. We could send any message at all to String, String just might not necessarily respond to that particular message. That’s why we have the respond_to? method, to see what messages an object responds to.

The messages an object will respond to need not be limited to the methods that are defined on that object. An object author can use the method_missing method to allow an object to respond to any message that’s sent to it, and respond in any way that the object author chooses.

Factory definitions and messages

What’s likely happening with first_name, last_name and email is that Factory Bot is using method_missing to allow arbitrary messages to be sent, and if the message (e.g. the message of first_name) matches an attribute on the model that the factory is for, then Factory Bot uses the block passed with the message to set the value of that attribute.

Takeaways

  • Factory Bot’s factory definitions are made out of methods and blocks.
  • Adding parentheses to any piece of DSL code can often make the code clearer.
  • Ruby objects can be passed arbitrary messages, and objects can be designed to respond to those messages.

Should I be doing test-driven development?

When I see questions from beginners regarding learning testing, sometimes they seem to conflate testing with test-driven development (TDD). People will say “I have such-and-such question about TDD” but really it’s just a question about testing, nothing to do with TDD specifically.

Other people sometimes ask questions about whether TDD is “better” than writing tests after.

In this post I’ll try to clarify what’s TDD and what’s not. I’ll also explain whether I think it makes sense for testing beginners to try to practice TDD.

Testing != TDD

First of all, at the risk of stating the obvious, testing and TDD aren’t the same thing. TDD is a specific kind of testing practice where you write the tests before you write the code that makes the test pass. (If you want to go deeper into TDD, I highly recommend Kent Beck’s Test Driven Development: By Example.)

Learning vs. incorporating

Another mistake beginners sometimes make is to conflate learning testing with incorporating testing as a habitual part of their development workflow. They feel like they need to start adopting testing practices into their workflow from day one, and if they fail to do that, then they’ve failed at learning testing.

I think it’s more productive to separate the jobs of learning testing and applying testing. It’s not like skiing, where you learn it and do it at the same time. It’s more like basketball, where you practice free throws in your driveway and build some skills that way before you try to play a real game in front of an audience. You’ll get farther in the beginning if you separate the practice from the application of what you’ve learned to production tests. When you get comfortable enough, you can take off the training wheels and get all your practice from writing production tests.

TDD is beneficial but optional

TDD is super helpful in certain scenarios but it’s not something you absolutely need to learn when you’re first learning testing. I think it’s completely appropriate to first learn the fundamentals of testing in general, and then start to learn TDD once you’ve developed a decent level of comfort with testing.

I don’t always practice TDD

I’m not an advocate of practicing TDD 100% of the time in Rails, even for experienced testers. The reason is that when I’m building a new feature, I often have little idea what shape that feature will take, and the most realistic way for me to hammer it into shape is to just start building it. Once I’ve built some of the feature, then I’ll start adding tests. So, a portion of the time, I write my tests after writing my application code.

The place where I find TDD most useful is for model code. I practice TDD in my models a high percentage of the time. Once I’ve put the broad strokes of a feature in place, I’ll usually use TDD to work out the fine-grained aspects of it.

Takeaways

  • Testing and test-driven development aren’t the same thing.
  • When you’re first learning testing, it can be helpful to separate learning testing from applying testing.
  • You don’t need to learn TDD when you’re starting out.
  • I don’t always practice TDD or even advocate practicing TDD 100% of the time. I myself practice TDD maybe 60% of the time.