Concurrent Acceptance Testing in Elixir

Posted on by in Development, Elixir

If you’ve practiced Test-Driven Development, you know that fast-to-execute tests are more than just a nice-to-have. As suites get slow, developers run them less often locally. Failures start to crop up in the CI environment, and the length of time between a breaking change and its detection increases.

The problem gets worse with acceptance tests. Since they execute in a browser, they’re slower to begin with and significantly more brittle. But with these negatives come a huge benefit: since they interact with your application in the same way a user does, they give you much more confidence that your system is working than isolated unit tests.

Running tests in parallel can sometimes help speed up tests, but doing so comes with its own set of issues. As with any concurrent code execution, global state can cause intermittent failures. In particular, very often tests rely on making changes to a database. These changes can easily leak from one test to another, causing havoc.

The upcoming release of ecto has a interesting new solution to this problem. It offers a connection pool built on the db_connection library, which provides a module named DBConnection.Ownership.

The ownership system

The idea is simple. Each concurrent test process must manually check out a database connection from a pool. When it does, it becomes the sole owner of that connection – no other process can access it. The connection is automatically wrapped within a transaction. Other concurrent tests won’t see any changes that ours is making to the database.

This works perfectly when all code is executing within one process, but, in acceptance testing, the test process and our web server are typically separate. To address this, the ownership pool offers something called allowances. An owned database connection can be used within another process as long as it is explicitly allowed. The two (or more) processes can then collaborate within a single database transaction, while all others are left blissfully unaware.

Tying it together

With the allow mechanism, we have everything we need to run concurrent acceptance tests. For these examples, we’ll be using the hound library, but the approach isn’t limited to it. We’ll start by writing a case template for our acceptance tests. Case templates let us share common code between multiple ExUnit tests.

We’re using Ecto.Adapters.SQL.Sandbox.checkout/2 to take ownership of a database connection. Then, we start hound and request a specific URL on our endpoint. We’ll use this request to pass information about our test’s repository and pid to the endpoint, so it can be allowed access to our database connection when responding to our requests:

We’ll need to configure our endpoint to listen for requests at this URL. We’ll do so by turning our module into a plug, and adding it to our Phoenix endpoint. We need to make sure to only do so in test. Although you might be tempted to write a conditional based on Mix.env in your endpoint, don’t. Mix is a not a run-time dependency, so it should only be used in development.

Our plug will handle requests at the url generated by path_for/2. We’ll stash the information the test process sends us in a cookie. The browser being driven by our acceptance test will store this cookie and present it back to the endpoint on subsequent requests.

Now that we’ve set the cookie, we’ll add another head to our plug’s call function. It will match any request not handled by the first one, so we can use it to allow access to our database connection.

Since we added this plug to our endpoint before our router, every web request to our controllers will pass through it. We fetch the information we need from the cookie, call allow/3 with the appropriate arguments, and we’re done!

Wrapping up

By parallelizing our acceptance tests, we can speed up the slowest part of our test suite, saving developers time and sanity on a daily basis. The Phoenix.Ecto.SQL.Sandbox plug is available in phoenix_ecto, as of version v3.0.0-beta.2. If you’re curious to see it in action, you can check out this example app. Note that this approach currently only works in selenium, because the PhantomJS driver shares cookies between its sessions, but support for phantom should be coming soon.

Special thanks to José Valim for the initial outline of the approach.