A flapping test is a test that sometimes passes and sometimes fails even though the application code being tested hasn’t changed. Flapping tests can be hard to reproduce, diagnose, and fix. Here are some of the common causes I know of for flapping tests. If you know the common causes of flapping tests it can go a long way toward diagnosis. Once diagnosis is out of the way, the battle is half over.
Race conditions
Let’s say you have a Capybara test that clicks a page element that fires off an AJAX request. The AJAX request completes, making some other page element clickable. Sometimes the AJAX request beats the test runner, meaning the test works fine. Sometimes the test runner beats the AJAX request, meaning the test breaks.
There’s a common race condition I’ve experienced in Capybara. A page will load, then Capybara will visit a subsequent page to verify some result. Unfortunately, the visit happens before the prior page gets a chance to fully do its thing, resulting in a failing test. An easy way to fix this is to add an expect(page).to have_content('some content') just before the visit. This way Capybara will wait for the first page to fully load before it tries to visit the second page.
“Leaking” tests (non-deterministic test suite)
Sometimes a test will fail to clean up after itself and “leak” state into other tests. To take a contrived but obvious example, let’s say test A checks that the “list customers” page shows exactly 5 customers and test B verifies that a customer can be created successfully. If test B doesn’t clean up after itself, then running test B before test A will cause test A to fail because there are now 6 customers instead of 5. This exact issue is almost never an issue in Rails (because each test runs inside a transaction) but database interaction isn’t the only possible form of leaky tests.
When I’m fixing this sort of issue I like to a) determine an order of running the tests that always causes my flapping test to fail and then b) figure out what’s happening in the earlier test that makes the later test fail. In RSpec I like to take note of the seed number, then run rspec --seed <seed number> which will re-run my tests in the same order.
Reliance on third-party code
Sometimes tests unexpectedly fail because some behavior of third-party code has changed. I actually sometimes consciously choose to live with this category of flapping test. If an API I rely on goes down and my tests fail, the tests arguably should fail because my application is genuinely broken. But if my tests are flapping in a way that’s not legitimate, I’d probably change my test to use something like VCR instead of hitting the real API.
I commonly review code committed with tests by my team that may work just fine locally, but in a CI/CD environment may fail and it’s usually the result API calls not finishing in time (case #1 in your post). My primary go-to is like you said: add some expectations around the changes expected. Rails has the lovely “#{Model} has been successfully updated.” statuses to help here, so adding an expectation for that has solved a ton of the flaky tests for me. Beyond that, in a remote CI/CD pipeline, I’ve found benefit in increasing Capybara.default_max_wait_time to make sure those assertions have time to get the response from the API. Just thought this may help some others in this case.