How to Test External APIs

Integrating with an external API is almost a guarantee in any modern web app. To effectively test such integration, you need to stub it out. A good stub should be easy to create and consistently up-to-date with actual, current API responses. In this post, we’ll outline a testing strategy using stubs for an external API.

Integrating with an External API

Our example app is a stock Rails 3.2 app. Let’s imagine that we’re assigned the following story:

This story involves integrating content from an external site. Hacker News doesn’t have an API, but they do have an RSS feed that we can use instead.

The Testing Strategy

A typical implementation of our story will look something like this:

  1. A request for the homepage is routed by Rails to a controller.
  2. The controller asks a Post model for recent posts.
  3. The Post model asks a Hacker News library for recent posts.
  4. The Hacker News library gets the latest posts from the Hacker News RSS feed.

To test this we’ll:

  1. Start with an end-to-end integration test. I like to use RSpec, so we’ll use RSpec Rails’ request specs. vcr will be used to record the actual Hacker News HTTP request and response.
  2. Skip controller specs. Directly specifying the controller won’t gain us much, because it will be thin and completely exercised by the request spec.
  3. Mock out the Hacker News library in the Post model’s specs. Since we already have an integration test, there’s no need to perform another mini-integration test between our domain model and its collaborator.
  4. Specify the Hacker News library directly. Instead of mocking, we’ll again use vcr to record the actual Hacker News HTTP request and response. This will give us the ability in the future to change our HTTP client without breaking our specs.

Starting at the Outside with a Request Spec

Our request spec uses capybara to simulate a homepage request. We tell vcr to record any HTTP requests by using its RSpec metadata support.

spec/requests/homepage_spec.rb

The first time this spec is run, vcr will record the actual Hacker News HTTP request and response and save it to the filesystem. Future spec runs will then reuse this recorded HTTP interaction. Since we don’t know the latest posts on Hacker News at the time this spec is first run, we only specify an expected DOM structure.

This failing spec guides us to the following controller and view implementation.

config/routes.rb

app/views/posts/index.html.haml

app/controllers/posts_controller.rb

The second run of the request spec goes further than the first, moving us to the domain model.

Mocking Collaborators in the Domain Model Spec

Instead of programming our controllers and views to the interface of our Hacker News library, we’ll introduce a domain model. The domain model is where we’ll consolidate our app’s business logic. It also gives us another level of indirection, allowing us in the future to change how we get Hacker News content without having to change our controllers and views.

Our domain model won’t directly make any HTTP requests. Instead, it will collaborate with a separate Hacker News library that will be responsible for making HTTP requests and parsing RSS. This library acts as an adapter layer, allowing our domain model to define its relationship to the outside world in its own terms. With an integration test already in place, we can use mocking to test our domain model in isolation.

spec/models/post_spec.rb

lib/hn/post.rb

app/models/post.rb

Our domain model is now specified but a re-run of our request spec reminds us we still have work to do. Mocking allowed us to implement our domain model by faking its collaborator, but we still have to implement that collaborator. Whenever you mock, you must have a higher-level test that actually tests the full integration between all your objects.

We can now drop back down and spec out the Hacker News library.

Black-Box Testing the Library

Our Hacker News library will be responsible for getting the latest posts from Hacker News. Like in our request spec, we’ll use vcr to record the actual HTTP request and response to Hacker News. This keeps our spec accurate. Future runs of this spec will then reuse the recorded HTTP interaction. This keeps our spec fast. Stubbing at the HTTP level instead of mocking out our HTTP client, gives us the ability to change our HTTP client later (granted it’s one of vcr‘s supported HTTP clients).

spec/lib/hn/post_spec.rb

typhoeus and Ruby 1.9.3’s RSS library work together to request and parse the Hacker News RSS feed.

lib/hn/post.rb

With our library specified, we can move back up to our request spec, which should now be passing.

Staying in Sync with External APIs

In our request and library specs, we used vcr to record an actual HTTP request to Hacker News and then in future spec runs reused that recorded HTTP interaction. The specs were accurate and fast but, if the Hacker News RSS feed changes, our specs still continue to pass. We need to keep our stubs up-to-date.

Fortunately vcr can be configured to automatically re-record “out-of-date” HTTP interactions.

spec/support/vcr.rb

vcr stores a timestamp alongside every recorded HTTP interaction. The above default_cassette_options configuration tells vcr to re-record all HTTP interactions that are older than 1 week.

Summary

Depending on an external API in your test suite can easily lead to tests that sometimes pass and sometimes fail. To make your tests run consistently, stub out external APIs. Ensure your stubs are accurate by creating them from actual API responses. And finally, remember to routinely verify that your stubs are up-to-date with the current state of the external API.

About Jared Carroll

After a short stint in the fashion industry Jared found his true calling at Carbon Five. Yes... he looks like a serial killer in this photo. But really he is as gentle as a flower.
This entry was posted in Process, Web and tagged , , , , , . Bookmark the permalink.

18 Responses to How to Test External APIs

  1. Avdi Grimm says:

    Excellent guide to The Right Way To Do It. Thanks for writing this!

  2. myronmarston says:

    I heartily approve :).

  3. Cyril Rohr says:

    Very nice, and to the point. VCR and WebMock are great gems for testing external HTTP APIs.

  4. Great post, Jared.

    I wonder what the best strategy for updating HTTP responses from VCR is. A few thoughts:

    * Updates have to happen automatically on a period (like the one week in your example), so that the test suite can survive people forgetting that they have to manually update it. It is important, in this scenario, that test failures coming from changes to the external API are verbose enough to let a new developer understand what just happened – since these failures will appear to happen completely out of the blue.

    * How do you handle committing changed HTTP response cached files to the repo? ie: what’s the git strategy for that? It is kind of weird that once per week (per external call being tested) some random developer sees a code change in his local repo that he didn’t make. Plus, if he’s in the middle of some feature branch, it’s sort of a pain to extract that change in to its own branch.

    * It seems that it might be a good idea to test the actual external API every time in acceptance. Is that a configuration option within VCR?

  5. Rob Pak says:

    Thanks for the post Jared,

    What is the best way of handling third-party API responses that require state (i.e. authentication and flow)? I found it brittle to re-record responses that required more than one interaction to produce the response that I needed. The RSS feed example is great but what about re-recording an API that requires three requests before the one that you need can be made.

    • Hi Rob,

      The RSS feed example does only illustrate using vcr to record a single HTTP request, but I’ve had success using vcr to record multiple HTTP requests as well. It does slow down the initial test run but the cassettes are re-used on future test runs.

      Once scenario that is somewhat ugly is when the API needs some initial state before you make a request to it. In those cases, I’ve had to resort to bootstrapping the API with the data that the tests need. However, if the API offers a way to create that data, then I’ll create it through the API. This is only possible of course, if you have access to the API.

  6. Very helpful,Thanks ! ! !

  7. Specs in your example only compare the structure of data, not the data itself. This is approach is limited to simple cases such as ‘get the latest posts’, but once you need something like ‘get latest posts that match user query’ you actually need to check that your app shows the right data in the end. How does vcr help here?

    So far, based on this post, the benefit of using vcr is in tests speed up and guarding against potential APIs outages. At the cost of more test setup (minor) and keeping stubbed data in sync (major). If that is accurate, than I am not convinced it is worth it.

    But I might be wrong.

    • The above example could be re-written to check the actual Hacker News content. It would have to check something that would be consistent each time the vcr cassette is re-recorded; e.g., it could verify that the published on dates of the items in the RSS feed are say, within the last week.

      In your example of a search query, you could use a similar strategy. If you searched for the word “ruby”, your tests could check that the items returned from the API have the word “ruby” somewhere, perhaps in their title. Testing for anything more would probably result it a brittle test that would need to be updated each time the vcr cassette is re-recorded.

      • So vcr is not intended for proper stubbing, but rather is more suited for testing assumptions about external services. E.g. ‘does this json really (still) contains that key and is its value still a datetime?’ sort of thing.

        • myronmarston says:

          Accurate HTTP stubbing is exactly what VCR is intended for. I think your question comes down to tradeoffs. You can write tests that assert exact results based on a recorded VCR cassette, but the tradeoff is that unless your assertion is based on an invariant that should always be true of the HTTP responses, the test will probably fail if/when you re-record the VCR cassette. But without VCR, the problem is even worse: if your test isn’t asserting on an invariant, the test will pass one minute, and then fail the next when the content of the HTTP response is different. The recorded fixture at least gives you consistent, deterministic results for as long as you’re willing to keep using that fixture.

          VCR works fine either way, you just need to decide what makes sense for your application.

          FWIW, this destroy all software screencast discusses this exact issue and has some good insight, I think:

          https://www.destroyallsoftware.com/screencasts/catalog/sucks-rocks-3-the-search-engine

          • Assertions based on invariant are limited and result in weak tests.

            Assertions based on variant mean (in vcr) you never rerecord cassette. So you might as well just use curl instead to prepare fixtures once.

            Imo, where vcr would actually be useful is for verifying those fixtures are still relevant. By checking that fixture metadata we care about match their up-to-date counterpart.

            It is like two separate layers. And it totally makes sense from bdd perspective: use fully-fledged mocks/stubs of external dependencies while testing domain and have separate (anti-corruption?) layer for testing your assumption about those dependencies.

  8. Chris St. John says:

    Done with care and style, I enjoyed this!

  9. Groovy!! I need this for my “final project” thank you

  10. Pingback: The Mega March 2012 Ruby and Rails News and Release Roundup

  11. sfsekaran says:

    Hey guys, this helps me a lot actually. Kudos :)

  12. Pingback: 5 reasons why database integration sucks « Pragmatic me

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>