Why I recommend against using Cucumber

by Jason Swett,

Around the time I first started using Rails in 2011, I noticed that a lot of developers, seemingly all Rails developers, were using Cucumber to assist with testing.

I bought into the idea—describing test cases in plain English—but in practice I found Cucumber not to be very valuable. In fact, my experience has been that Cucumber adds a negative amount of value.

In recent years I’ve noticed (although this is anecdotal and might just be my perception) that fewer codebases seem to use Cucumber and that fewer Rails developers seem to be on board with Cucumber. I had thought Cucumber was pretty much dead. But lately, to my surpise, I’ve seen Cucumber recommended to testing noobs more than a few times. Since I consider Cucumber to be a bad thing, I want to explain why I think so and why I don’t think other people should use it.

In my view there are two general ways Cucumber can be used: it can be used as intended or it can be abused. In the former case, I believe Cucumber has a small negative value. In the latter case I believe it has a large negative value.

Why Cucumber is bad when it’s not used as intended

Most production Cucumber scenarios I’ve seen look something like this:

Given a user exists with email "test@example.com" and password "mypassword"
And I visit "/sign_in"
And I fill in the "Email" field with "test@example.com"
And I fill in the "Password" field with "mypassword"
And I click "Sign In"
And I visit "/user/edit"
And I fill in the "First Name" field with "John"
And I fill in the "Last Name" field with "Smith"
And I fill in the "Age" field with "30"
And I click "Save"
And I visit "/profile"
Then I should see "John Smith, 30"

These kinds of tests, with fine-grained steps, arise when the developers seem to mistake Cucumber for a way to write Ruby in English. The above scenario provides exactly zero benefit, in my opinion, over the following equivalent Capybara scenario:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The Cucumber/Gherkin version is no shorter nor more easily understandable.

To be fair to Cucumber, nobody who understands Cucumber advocates writing Cucumber scenarios in this way. The Cucumber creator himself, Aslak Hellesøy, wrote a post in 2011 saying not to do this. Other people have written similar things.

I think it’s telling that so many people have written blog posts advising against the very common practice of writing fine-grained Cucumber steps. To me it’s kind one of those gas station doors that looks for all the world like a pull door, so every single person who comes up to it pulls it instead of pushes it, feels like a dumbass, and then pushes it. So the gas station manager puts up a big sign that says “PUSH”, but most people don’t notice it and the problem persists. What instead should have been done is to make the push door look like a push door, without the big useless handle that you’re not supposed to pull. I get that the Cucumber maintainers tried to do that by removing `web_steps.rb`, but in my experience it didn’t seem to work.

And it doesn’t matter much anyway because Cucumber still sucks even if you don’t abuse it by writing fine-grained steps. I’ll explain why I think so.

Why Cucumber is bad even when it is used as intended

Here’s a version of the above Cucumber scenario that’s done in the way the Cucumber creators would intend. There are two parts.

First, the Gherkin steps:

Given I am signed in
And I provide my name and age details
Then I should see those details on my profile page

Second, the underlying Ruby steps:

Given /^I am signed in$/ do
  visit sign_in_path
  fill_in 'Email', with: 'test@example.com'
  fill_in 'Password', with: 'mypassword'
  click_on 'Sign In'
end

And /^I provide my name and age details$/ do
  visit edit_user_path
  fill_in 'First Name', with: 'John'
  fill_in 'Last Name', with: 'Smith'
  fill_in 'Age', with: '30'
  click_on 'Save'
end

Then /^I should see those details on my profile page$/ do
  visit profile_path
  expect(page).to have_content('John Smith, 30')
end

This is actually pretty decent-looking and appealing, at least at first glance. There are two problems, though. First, this way of doing things doesn’t really provide any clarity over doing it the Capybara way. Second, the step definitions usually end up in a single, flat file full of “step soup” where unrelated steps are mixed together willy-nilly.

Compare this again with the Capybara version:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

visit sign_in_path
fill_in 'Email', with: 'test@example.com'
fill_in 'Password', with: 'mypassword'
click_on 'Sign In'

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

The sign in portion is usually abstracted away in Capybara, too, so the scenario would look more like this:

FactoryBot.create(:user, email: 'test@example.com', password: 'mypassword')

sign_in

visit edit_user_path
fill_in 'First Name', with: 'John'
fill_in 'Last Name', with: 'Smith'
fill_in 'Age', with: '30'
click_on 'Save'

visit profile_path
expect(page).to have_content('John Smith, 30')

That’s not too crazy at all. In order for Cucumber to be a superior solution to using bare Capybara, it would have to have some pretty strong benefits to compensate for the maintenance burden and cognitive overhead it adds. But it doesn’t.

So what do I recommend doing instead of using Cucumber? I think just using Capybara by itself is fine, and better than using Capybara + Cucumber. I also think Capybara + page objects is a pretty good way to go.

18 thoughts on “Why I recommend against using Cucumber

  1. Romeu Fonseca

    Cucumber is useful when the story carries all and only the business information, keeping the test implementation far from it.

    As an example, when a big refactor is necessary, is very easy to have rspec and capybara left with no business rules and a bunch of tests tied to the implementation as a dead weight.

    Reply
    1. sfcoding Post author

      I can see the value in that. On the other hand, as a principle, I never do big refactors. I only ever do refactoring a little at a time.

      Reply
  2. Konstantin Gredeskoul

    I completely agree with you, and have been pretty much ignoring cucumbers since 2012 (been doing ruby since 2005).

    For me cucumber is too much “magic”.

    I don’t care to look for regular expressions that map to cucumber steps. I’d rather see what is actually going on.

    I also am annoyed that newly generated Rails apps by default have Cucumber tests. Have you ever tried explaining how that works to someone new to Rails? It’s a disaster.

    Thanks!

    Reply
  3. Lucas Luitjes

    In my opinion there is one exception: when the Cucumber features double as technical documentation, for people who are not necessarily familiar with the implementation language/framework. In that case the benefit this pretty huge, because you have documentation that is, never, ever oudated.

    For example: for a front-end developer, well written cucumber features can be executable documentation for a REST API. I’ve seen that work rather effectively for SPA’s where backend and front-end development was done by separate teams. Simplified code example:

    Given my database contains following users:
    | id | username |
    | 1 | Moriarty |
    When I GET “/users/1” with content type “application/json”
    Then I expect the following JSON:
    “””json
    {
    “id”: 1,
    “username”: “Moriarty”
    }
    “””

    Another example: a CLI tool. In that case the Cucumber features are executable examples:

    Scenario: do not allow faulty arguments
    When I run `my_cli_tool –faulty-args`
    Then the exit status should be 1
    And stderr should contain:
    “””
    Please do not use –faulty-args. Instead, use –non-faulty-args. See full docs by running `my_cli_tool -h`
    “””
    And stdout should be empty

    But I agree if the only consumers of your scenarios, are people who can easily read Ruby/rspec, there’s not any added value.

    Reply
    1. Jason Swett Post author

      “But I agree if the only consumers of your scenarios, are people who can easily read Ruby/rspec, there’s not any added value.”

      I’ve never worked anywhere where non-Rubyists needed/wanted to read the specs. I’m certainly not saying that case doesn’t exist. I know it does. What I’ve seen more commonly though is developers assuming that non-technical people will want to read or write Gherkin, only to discover after a lot of work that that’s not the case.

      Reply
  4. Steven Webb

    I agree with with you Jason. Personally I find by using methods I can get very readable specs. E.g.,

    “`
    scenario “successfully updating profile details”
    FactoryBot.create(:user, email: ‘test@example.com’, password: ‘mypassword’)
    sign_in

    provide_name_and_age

    visit profile_path
    expect(page).to display_profile_details
    end

    def provide_name_and_age
    visit edit_user_path
    fill_in ‘First Name’, with: ‘John’
    fill_in ‘Last Name’, with: ‘Smith’
    fill_in ‘Age’, with: ’30’
    click_on ‘Save’
    end

    def display_profile_details
    have_content(‘John Smith, 30’)
    end
    “`

    Reply
  5. Ronaldo

    I agree in part with Jason. In my case, the stories and BDD (feature files) are written by the system analysts that has the knowledge of business role. A feature is used to run end to end flows in many environments, just changing the values.
    Another think to consider is the reuse of the steps. If you build a good architecture to support a automation project, Cucumber will help you to document and separate the business roles of your code.
    Cucumber features is easily written by any analyst, not technical skills necessary also we can control the version of any changes on business roles.

    Reply
  6. Oli Sawtell

    Honestly, your missing the point, Cucumber and Gherkin syntax is used to ensure that everything is readable to everyone in the project – not JUST testers and developers. Now I have taught our clients how to write ‘cases’ using gherkin syntax, it is a simple matter for me to take what they make, create the steps in Ruby (I prefer Watir over Capybara – but the underlying framework isn’t relevant to this discussion) and fire off the steps. Asking a client to write a test case in Capybara? no chance!! Cucumber is about BDD and good comms.

    Reply
  7. Rich

    Generally agreeing with the “abuse” stuff in your article and also with the people who say that Gherkin is a great tool for communicating acceptance tests / specifications with non-technical customers – something I have to do all the time. They can sign off on the acceptance tests in Gherkin format, and can be used equally well for the automated tests (for my benefit) and the manual demo (for the client’s benefit).

    For the final point about “step soup” – I’m in total agreement. Thankfully, Spinach exists as an alternative Gherkin runner with scoped steps – if you want steps that are common to multiple features, you have to explicitly declare them as such and include them. It even has an audit command to check your step files are clean and complete.

    Reply
  8. David Johansson

    At my last QA job it was so frustrating that all interviewees tended to say how awesome cucumber was – I suspect it is because it’s easy to find into stuff for cucumber but in my experience no one seems to have best practices with there instructions. Cucumber has never been a tool I would use because I can always find better solutions.

    Reply
  9. Pingback: 週刊Railsウォッチ(20180723)Railsdm Day 3 Extremeを後追い、PSDにはZeplin.io、好みの分かれるJSX、負荷テストツール比較ほか

  10. Alex C

    Cucumber is trying to solve the problem of making 100 equal to the number 1. “Oh it’s simple, here’s the requirement in plain ” . Product and other nontechnical people have to deal with edge cases, unexpected discoveries in their product, unthought of use cases and other things that show up just like engineers do. Just because it looks nice in “plain “, doesn’t mean it’s useful. Listing out all of those exceptions, surprises, and hidden scope will NEVER be solved, or even helped by a tool like Cucumber. If you need product specification, you need product specification. Hiding details hides all of the important things. In fact it’s worse because you’re using a LL1 parseable language to describe these hidden things. Do non technical people enjoy reading and writing code that pretend to be requirements?

    Reply
    1. b.ben

      The first time I read your article I totally agree with you.

      But I just come back and rethink.

      Cucumber is not the main to write requirement, It’s going to be eye pain for devs to read it.

      And it’s totally useless for businesses person to read Cucumber later.

      It’s only totally useful when we do regression and requirement guys to ensure that features still work.

      I think we need to see cucumber as a tool to hide the implementation details.

      To let PO or BA writing it without touching the actual code then having compile error.

      But if you’re a dev who going to write those tests, Cucumber is useless.

      Reply
  11. k.el

    Sorry to let you know! But Rspec itself is tested using Cucumber.
    One of the great things out there is that you can read the documentation on Relish which is generated from the feature files (the actual behaviour, no more, no less).
    I think you are missing the whole point about Cucumber. Describing the behaviour, not implementation details.

    Also, you are saying that Cucumber has one file with unrelated steps. That is not true at all.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *