Test Suite Health Check

Test Suite Health Check

Dear developer or leader of developers:

There are about three possible states for your company’s test suite to be in.

  1. Good enough
  2. In need of improvement
  3. Nonexistent

If your test suite is “good enough”, then you’re probably in the top 1% of organizations when it comes to testing. You can stop reading here because the rest of what I’m going to write won’t interest you.

If’s far more likely that your test suite is aching for improvement and incurring unnecessary development costs. In this case I have a message for you.

If you don’t have a healthy test suite, you could be suffering these expensive consequences

I’ve worked at a number of places where development was limping along at a fraction of the speed it could.

What was worse is that although leadership was aware that there was a problem, they were blind as to how to fix it because they believed their development practices (and related pains) were normal and inevitable!

Here are some issues I’ve seen arise as a result of not following healthy testing practices (and other related practices):

  • Deployments are risky, nerve-wracking “shuttle launch” type affairs
  • Deployments are often followed by firefighting and damage control
  • Features are only released to users every 2 weeks, 4 weeks or even worse
  • Features get released that contain bugs, causing rework
  • Manual QA consistently finds bugs in “finished” work, causing rework
  • Developers play “whack-a-mole”, fixing one bug only to introduce another
  • Developers’ estimates are routinely off by a factor of 2/3/4/10
  • No one trusts the development team because they never finish their sprints

What if all these problems went away?

Imagine the opposite scenario from the one above:

  • Deployments are a non-event that no one notices
  • Customers get new features and bugfixes almost every day
  • Newly-released features just work
  • Manual QA only uncovers bugs for nuanced edge cases
  • Bugs that get fixed stay fixed
  • Developers provide estimates that are usually realistic
  • Stakeholders trust the development team to fulfill promises

You might be reading this and thinking, “Testing can’t fix all this stuff! Plus you’re just talking about DevOps principles.”

Both those things are true. Testing by itself can’t fix all these things – but when healthy testing practices and a healthy test suite are combined with certain other practices, testing can help fix a lot of these things.

“How do I know if my test suite sucks?”

These are some common symptoms of an unhealthy test suite:

  • The test suite often has to be re-run multiple times to pass due to “flapping tests”
  • Test failures are often ignored due to the desensitizing effect of flapping tests
  • The development team spends a lot of time wrestling with test code
  • Writing tests is often skipped because the test code is so painful to work with
  • Test cases are confusing and full of implementation details
  • Test code is repetitive and clunky
  • The test suite is run infrequently because it takes so long to run

In contrast, in a healthy test suite:

  • One run of the test suite is enough because the test suite is deterministic
  • Test failures are always heeded as indicators of genuine problems
  • Tests are always written with relative speed and ease
  • Test cases are abundantly clear and understandable
  • Test code is concise and elegant
  • The test suite is fast and can be run frequently

How a Test Suite Health Check works

Right now I’m offering a service called the Test Suite Health Check. Here’s how it works.

  1. You pay the service fee and give me access to your repo(s)
  2. We’ll get on a call so you can “tell me where it hurts” and I can ask you some questions about your current testing practices
  3. I’ll examine your test code
  4. I’ll deliver to you a Test Suite Roadmap that you can use to improve your test code
  5. We’ll get on a call to go over the Roadmap

Will this service solve all your problems? Of course not. Turning around an unhealthy testing situation can take months of dedicated work. But the Test Suite Health Check can at least show you where your opportunities for improvement lie and get you pointed in the right direction. (And by the way, if you decide you want more help beyond the Health Check, I’m here for that too.)

Cost and Application

The fee for the Test Suite Health Check is $650. To apply just fill out the form below.

After you submit your application I’ll follow up to schedule a call to see if we’re a good fit.