An important principle of testing is that tests should be deterministic. The passing or failing of a test shouldn’t depend on the date it was run, whether any certain other tests ran before it, or whether some condition outside the application changed.
If a test hits an external URL then it’s susceptible to being non-deterministic. I’ll give you an example from something I was working on today.
Today I wrote a test that said, “When I save a new podcast to the database, all the episodes listed in that podcast’s XML feed should get saved to the database.” I pointed my code at the XML feed URL for the Tim Ferriss Show and expected that when I saved the Tim Ferriss Show, 349 episodes would go into the database.
You can probably sense the problem with this test. What happens when I run this test next week when Tim Ferriss has 350 episodes? The test will fail even though the code will still work.
Basically I changed the level of responsibility of the method that parses the XML feed. The original “agreement” was “I’ll give you an XML feed URL and you grab its contents from the internet and parse the file and save the episodes to the database”. I changed that agreement to “I’ll give you the contents of an XML file (which could have come from the internet or from the filesystem) and you parse it and save the episodes to the database”.
Here’s what my parsing method looks like:
This method (which you can see in context on GitHub here) doesn’t know or care where the XML feed contents came from.
This means that in production I can pass XML feed contents that came from the internet, and in test I can pass XML feed contents that came from the filesystem.
Does this mean that the actual act of downloading the XML feed content from the internet is untested? Yes. That’s a downside, but I think it’s a small one and I prefer that downside over the downside of having non-deterministic tests.