Testing Anti-Pattern: Setup Data Leak

by Jason Swett,

In general it’s better to have tests than to not have tests. Most people agree on this.

But just like poor-quality code, poor-quality tests get hard to maintain over time.

One anti-pattern I see in testing, an anti-pattern that makes tests harder and harder to maintain as the tests grow, is what I’ll call “setup data leak”. Here’s what it looks like.

Let’s imagine we have a User class. Maybe the user class can do things related to accounts, roles and posts.

In order to test accounts, roles and posts, we have to create some account, role and post data.

A Bad Test

Why is this test bad? Because it defines all the test data in one place. Why is that bad? Two reasons.

The “Single Glob of Data” Phenomenon

Let’s imagine that this test file grows over time and instead of setting up 4 things at the top, we set up 15. How could anyone be sure which of those 15 things is needed for any particular test? Maybe some tests only need 3 of those 15 things. Maybe other tests need all 15. Maybe still others need 8 of them.

Since we don’t know what’s needed for any particular test, we basically have to assume that everything is needed for every test until we prove otherwise on a case-by-case basis. So our 15 things aren’t 15 things anymore but one single horrifying glob of data.

Mystery Guests

Let’s again imagine that this test file grows over time. Deep within the bowels of one particular test you see role. What’s role? You look to the immediate vicinity and find nothing. Eventually you see that role is defined at the very top of the file even though it’s only used in one place.

A mystery guest is a value that appears in a test case but is defined far away from the test case.

If a value is truly used everywhere then there’s no reason not to define it at the very top of the test. But if it’s only used in one place, there’s no reason not to define it right next to the place it’s used.

A Better Test

This test is better than the first one because there’s no “setup data leak” issue. All the setup data is defined close to where it’s needed. To put it another way, all the setup data is defined in the narrowest scope possible.

Did you find this article useful? See all my Ruby/Rails testing articles.

Leave a Reply

Your email address will not be published. Required fields are marked *