Ruby Testing Glossary

Ruby-Specific Terms

Capybara
Integration testing tool. Capybara’s GitHub page describes it as an “acceptance test framework for web applications” but everyone I know uses the term “integration test” when they talk about Capybara. I would even argue that this is a misapplication of the acceptance test term.
Cucumber
A tool that lets you write test scenarios in English commands and then tie those English commands to code. For example, a Cucumber file could contain the line Given I am on the login page. In a different file, the text Given I am on the login page would map to some Ruby code that navigates to the login page.

I personally advise people not to use Cucumber. This article by Kevin Liddle does a pretty good job of articulating the reasons why I think using Cucumber isn’t a very good idea.

Database Cleaner
The tests in a test suite should be runnable in any order. If a test leaves behind extra data or depends on data set up in a previous test, there’s a good chance the ability to run the tests in any order has been lost. Database Cleaner is a tool that helps ensure every test case starts and ends with a blank database.
factory_bot
“A library for setting up Ruby objects as test data”, according to factory_bot’s GitHub page. I think that’s a pretty good description. If I were to say it in my own words, I’d describe factory_bot as a library that helps define factories for the purpose of conveniently creating test data.
Faker
Faker is, for my purposes, a tool that generates random values so I don’t have to manually come up with arbitrary phone numbers, names of people, etc. when testing. Faker is also handy when I need test values to be unique, e.g. when having two people with the same name would be a problem. I use Faker in conjunction with factory_bot.
MiniTest
A Ruby testing framework.
RSpec
A Ruby testing framework. In my perception, RSpec is by far the most popular testing framework in the Rails community.
System Tests
A feature added to Rails core in Rails 5.1 to make integration testing easier. Uses Capybara. If you’d like to learn more about System Tests, Noel Rappin wrote a pretty good Rails System Tests guide.
Test::Unit
A Ruby testing framework.
timecop
A library that makes it easy to test time-dependent code. Offers features like “time travel” and “time freezing”.

Non-Ruby-Specific Terms

Acceptance Test
A test that checks whether a feature meets its business requirements. A certain feature may work exactly as the developer and designer intended, but if the developer and designer designed and built the wrong thing, the acceptance test would fail.
Assertion
A condition that evaluates to true if the feature behaves properly and false if the feature contains a bug. The analogous term in RSpec is “expectation”.
CI Server
A server that aids in the practice of continuous integration. A CI server typically runs the test suite anytime a code change is pushed and notifies the development team in the case of a test suite failure.
CircleCI
A CI server product.
Continuous Integration
The practice of continuously merging together the work of all the developers on a development team. The benefit of continuous integration is that it helps teams avoid the painful process of integrating large chunks of work, which is often tedious and error-prone.
Dependency Injection
A technique where an object’s dependencies are passed as arguments, making it easier to test objects in isolation. James Shore defined dependency injection as simply “giving an object its instance variables”. I like that definition.
Factory
There’s a design pattern called the Factory Pattern or the Factory Method Pattern and it exists independently of anything to do with tests. I basically think of a factory as code that spits out an object of some sort.

I included the Factory term here because factories are very relevant to how I write tests. Rather than manually spinning up a bunch of arbitrary data using fixtures, I prefer to define factories that can hide away irrelevant details, allowing me to focus only on the parts I care about. For example, if I want to create a restaurant for testing purposes, I might not want to have to specify the restaurant’s phone, address, hours, or any of those other details, even though they may be required attributes. Most of the time I just want to specify the restaurant’s name and nothing else. Factories can make this job very easy and convenient.

Fixture
A way of setting up test data. In Rails, fixtures are defined using YAML files. I prefer factories over fixtures for a number of reasons. One of the main reasons is that I find fixtures to be too “distant” from the tests, making it somewhat mysterious what data is available for any particular test.
Flapping test
A test that sometimes fails and sometimes passes, seemingly at random. Also known as a “flickering” test. They’re often a symptom of one test case “leaking” into subsequent test cases and affecting their behavior.
Edge Case
A path through a UI that isn’t exercised frequently or that is unlikely to be exercised.
End-to-End Test
A test that exercises all the layers of an application stack. It could be said that an end-to-end test is more inclusive than an integration test and that an integration test only has to combine two or more layers of an application stack to be considered an integration test. In practice I’ve found that people use the terms “integration test” and “end-to-end test” fairly interchangeably and if you want total clarity, you have to ask what exactly the person means.
Environment Parity
Environment parity is a broader version of the dev/prod parity term from the Twelve-Factor App principles. The idea is that every difference between two environments is an opportunity for a bug to be present in one environment that is not present in the other. For this reason it’s helpful to keep the test, development and production environments as similar to each other as possible.
Happy Path
A normal/default/valid path through a feature. (I like to think of the opposite as the “sad path”.)
Integration Test
A test that combines two or more layers of an application stack, or that combines two or more systems. See also: end-to-end test.
Mock
A “fake” object. A mock is useful when you want to test a certain object, but don’t want to deal with the hassle of bringing that object’s dependencies into the picture. The dependencies can instead be mocked.
Regression Testing
A type of testing meant to ensure that previously-working functionality has not been broken by newly-developed functionality.
Selenium
A tool for automating browsers. Also, a chemical element.
Stub
A term that may or may not mean the exact same thing as mock. Mock and stub are not terms that seem to have crisp industry consensus.
System Under Test
The system that’s being tested.
Test Case
I don’t think I can improve on Wikipedia‘s definition: “A specification of the inputs, execution conditions, testing procedure, and expected results that define a single test to be executed to achieve a particular software testing objective”.
Test Coverage
The percentage of an application covered by automated tests.
Test Environment
An independent instance of an application used specifically for the purpose of testing.
Test Suite
A collection of tests. An application might have one or many (or zero!) test suites.
Test-Driven Development
The practice of developing features by first writing tests that specify the behavior of the desired functionality, then writing the code to make those tests pass.
Unit Test
A fine-grained test that tests a very specific piece of functionality in isolation. Note: what Rails developers often refer to “unit tests” are not technically unit tests since those tests involve the database.

One thought on “Ruby Testing Glossary

  1. Mike

    Hello,
    Thank you for doing this summary. The one nit I would suggest is in your reference to the article on why Cucumber is not a good option. It may not be a good option sometimes, but this article glosses over the primary benefit of something like Gherkin. It provides for the ability to specify how your application works in a language that is significantly easier to understand and significantly easier to change than your code. If you already know what you are going to build and why, this may not be a big deal. But if you are building something, have someone give you a rough idea of what it is. Writing it in English (or your native language) so you think through how it has to work, is a huge benefit. If it is reviewed by someone else, such as a PM or a stakeholder, even better. It clearly communicates what you intend to build (and hopefully why). If it needs to change, much easier to change. That it can also work to run acceptance testing makes it nice, but not unique, for that need. In other words, to write an acceptance test, you should know what acceptable is. The gherkin syntax is good for that.
    Best,
    Mike

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *