Writing Reliable iOS Tests

Tests help me write better apps. Writing tests informs my interface designs, expresses some of my intentions, and guards against regressions. As applications grow so do the number of tests I’m running as a regular part of my development workflow. If I’m not careful those growing test suites can slow down, become inconsistent, and eventually lose the trust of the development team. Fortunately, test driving software design is not a new idea and we can look to other languages and frameworks with good testing practices for inspiration on how to avoid pitfalls we encounter when writing tests.

I ran into a couple of cases on recent projects where I wrote unreliable iOS XCTests. Let’s take a look at what went wrong, what a better test might look like, and what tools we, as iOS developers, might be missing.

These are BDD-style specs written using Kiwi. It runs as XCUnit test cases so the same ideas apply if you prefer to write xUnit style test methods.

Shared state

This test passed for weeks until a test for an unrelated class was added to the app and this test started failing. Why? Our tests did configure CoreData to use an in-memory store but never reset it’s contents between test cases. These tests leaked and inherited global state. When some other test created an unread Message and ran before this test it triggered a false negative test failure. Additionally the tests depend on the order in which they are runso a change as simple as a rename could alter their results.

This is a bad test but not a problem unique to iOS. Ruby/Rails testing tool like RSpec include two features which dramatically mitigate our ability to write such fragile tests.

1. The order in which tests are run is randomized on every invokation of the test suite. (Using a seed which is output in the test log and can be supplied via a --seed flag to force a stable ordering to debug issues.)
2. Each test is wrapped in a database transaction which is rolled back when the test finishes.

Randomizing the test order helps identify any leaking state early. It might be possible to implement the same in XCTest but I have not dug into how the test runner sorts test cases.

We can reset our CoreData managed object contexts or persistent stores inbetween tests using an XCTestObserver class. Since there’s no framework provided convention for how or where a CoreData stack is initialized this needs to be re-implemented for each app but model persistence libraries could work to make the task easier. For example RestKit provides a test factory class which can reset the managed object contexts and singletons it creates.

CoreData is not the only location where a test could leak state. GCD queues, keychain values, user defaults, files on disk, singleton classes, and other locations specific to an app might also need to be reset between tests. As authors of apps and libraries which include such behaviors we should make sure this is possible in order to write reliably testable code.

Reseting app state between test cases can also have a significant performance cost. I have found it useful to limit this to a functional test suite which tests interactions between systems (including persistence and queries against Core Data). My unit tests do not include such a reset but given that they should be stubbing any dependencies of the class under test a unit test should not introduce or depend on such shared state.

Depending on time

I was testing some presentation behavior something like:

This test would occasionally fail, displaying “2 days ago” instead of “yesterday” when a second rolled over during the test run causing oneDayAgo to then be 2 calendar days ago compared to the current time when the formatter performed its comparison.

Controlling the flow of time during a functional test is often useful and tools like Ruby’s timecop gem provide a nice interface for doing so. Unfortunately stubbing NSDate without causing XCTest to hang proved to be difficult. Instead it is easier to supply dates, including those we expect to be now, as parameters to methods which require them.

Depending on external services

Tests should not depend on remote services being responsive or network connections being available. Nocilla will intercept HTTP requests allowing us to provide stub responses and alerting us of any unanticipated requests. By failing early on any unexpected request we can avoid discoving accidental external dependencies later when the service we rely on proves to be unreachable. That’s a good solution for HTTP requests but apps may need other tools to stub other connection types. Setting explicit expectations on network requests when writing tests can also serve as a reminder for developers to consider how the app handles failures of those connections.

What pain points have you encountered when writing iOS tests? Are there tools from other languages you miss when working on this platform?