Sometimes it’s better for tests to hit the real API

by Jason Swett,

A mysteriously broken test suite

The other day I ran my test suite and discovered that nine of my tests were suddenly and mysteriously broken. These tests were all related to the same feature: a physician search field that hits something called the NPPES NPI Registry, a public API.

My first thought was: is something wrong with the API? I visited the NPI Registry site and, sure enough, I got a 500 error when I clicked the link to their API URL. The API appeared to be broken. This was a Sunday so I didn’t necessarily the NPI API maintainers to fix it speedily. I decided to just ignore these nine specific tests for the time being.

By mid-Monday, to my surprise, my tests were still broken and the NPI API was still giving a 500 error. I would have expected them to have fixed it by then.

The surprising root cause of the bug

On Tuesday the API was still broken. On Wednesday the API was still broken. At this point I thought okay, maybe it’s not the case that the NPI API has just been down for four straight days with the NPI API maintainers not doing anything about it. Maybe the problem lies on my end. I decided to do some investigation.

Sure enough, the problem was on my end. Kind of. Evidently the NPI API maintainers had made a small tweak to how the API works. In addition to the fields I had already been sending like first_name and last_name, the NPI API now required an additional field, address_purpose. Without address_purpose, the NPI API would return a 500 error. I added address_purpose to my request and all my tests started passing again. And, of course, my application’s feature started working again.

The lesson: my tests were right to hit the real API

When I first discovered that nine of my tests were failing due to a broken external API, my first thought was, “Man, maybe I should mock out those API calls so my tests don’t fail when the API breaks.” But then I thought about it a little harder. The fact that the third-party API was broken meant my application was broken. The tests were failing because they ought to have been failing.

It’s been my observation that developers new to testing often develop the belief that “third-party API calls should be mocked out”. I don’t think this is always true. I think the decision whether to mock third-party APIs should be made on a case-by-case basis depending on certain factors. These factors include, but aren’t limited to:

  • Would allowing my tests to hit the real API mess up production data?
  • Is the API rate-limited?
  • Is the API prone to sluggishness or outages?

(Thanks to Kostis Kapelonis, who I interviewed on The Ruby Testing Podcast, for helping me think about API interaction this way.)

If hitting the real API poses no danger to production data, I would probably be more inclined to let my tests hit the real API than to mock out the responses. (Or, to be more precise, I would want my test suite to include some integration tests that hit the real API, even if I might have some other API-related tests that mock out API responses for economy of speed.) My application is only really working if the API is working too.

2 thoughts on “Sometimes it’s better for tests to hit the real API

  1. Nishchal

    Well, that sounds wrong, there are different things to consider here, first the API is doing something weird, it’s not supposed to send you 500, but rather something else, I generally throw a BadRequest or PreconditionFailed.

    Second, it’s better to mock out the API, and if they have a breaking change, the endpoint should simply reflect that, either using header or by adding a v{n} to api endpoint. The API is at wrong end here, it’s not supposed to break other applications. But they did, but it’s not YOUR responsibility to test their API now, is it? If that were the case, we’d have to test someone else’s class which you installed using package manager as well, maybe they have a breaking change but didn’t follow semantic release?

    Reply
  2. Matthew Puku

    Nishchal — you’re right that the problem is more on the API’s end than on Jason’s app. But I think it’s healthier to take an attitude of “the buck stops with me” when you’re dealing with external APIs, and do whatever it takes to deliver a good outcome for your users. If that means testing things that aren’t “my responsibility”, then so be it!

    Packages are a little different. Say you’re using Rails’s presence validator to prevent appointments with no dates. Great! Just test that users can’t create appointments without specifying a date! There’s no need to directly test the presence validator. And there’s also no need to mock it.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *