All code eventually gets tested, it’s just a question of when and how and by whom

by Jason Swett,

All code eventually gets tested. If a bug is found, it can be found in one of the following ways, in descending order of cost and embarrassment.

A Customer Trying to Use the Feature in Production

Obviously the worst place to find a bug is in production, and the worst person to find it is an end user trying to do his or her job.

This is a bad way to find a bug not just because it blocks your user from using your feature successfully and thus hurts your reputation, but because you might not even immediately find out that the bug exists. Not every user who finds a bug immediately reports that bug to the vendor’s development team. When I find a bug in Gmail, for example, I don’t make an attempt to report the bug to the Gmail team, I just suffer through it.

Even if the customer reports the bug, the bug probably won’t get directly reported to you, the developer. It will probably get reported to a support person, then get communicated to your manager, then communicated to you. By this point the bug may gain high visibility inside your organization, providing embarrassment to you and your team.

Discovering bugs in production is of course inevitable but all possible measures should be taken to avoid it.

Your Boss/Client Trying to Use It

The next worst way to find a bug is for your boss or client to find it.

I don’t mean when your boss/client serves in a QA tester role. I mean when your boss/client tries to use the feature with the expectation that it’s already finished and verified by you to be working.

The reason that this is bad is obvious: it erodes the trust your boss or client has in your competence as a developer. It especially hurts to get back a comment like, “Hey, let me know when this is 100% working.”

Not only is there the embarrassment and trust erosion but a concrete cost as well. The back-and-forth between you and your boss/client lengthens the development cycle for the feature, and therefore increases the cost to develop that feature.

Manually Testing it Yourself

Catching a bug by manually testing the feature yourself is way better than letting your boss, client or end user find it.

If you discover a silly bug, you might still feel a little embarrassed, but it will be a small, private embarrassment. You also keep the development cycle shorter by skipping the back-and-forth of “Hey boss, this is done,” “Hey developer, it’s not actually done.”

An Automated Test

The least expensive way to catch a bug is to write an automated test that makes the bug physically impossible.

Some developers perceive that writing automated tests takes “extra” time. That’s an inaccurate way to look at it. All testing takes time. The difference is that with a manual test, the time is paid each time you carry out the test, and with automated testing the time is paid all at once upfront. It’s like the difference between paying cash for a $250,000 house versus getting a mortgage and paying $500,000 for the same house over the course of 30 years. It potentially only takes a small handful of runs of a manual test before you’ve used up more time than it would have taken to write an automated test.

“So You’re Saying Automated Tests Are The Answer?”

My point is not that automated tests are a better alternative to all the other forms of testing. My point is that I believe automated tests are, in most cases, the least expensive testing method and the best first line of defense.

Imagine a secret government research facility that’s protected by an armed guard, locking doors and safe containing sensitive documents. If a Russian spy is after the documents, it’s obviously better for the spy to get caught by the guard than for the spy to make it past the guard and locked doors and make it all the way to the safe. But that doesn’t mean it’s a smart idea to do away with the doors and safe and only have the guard. Defense should exist at all levels practicable.

Same thing with automated testing. Just because it’s cheaper for a bug to get caught by an automated test doesn’t mean it’s a good idea to do away with manual testing. Protection should exist at all levels. It’s just a good idea to try to catch as many bugs at the earliest levels as possible.

Leave a Reply

Your email address will not be published. Required fields are marked *