Getting “Test”-y in iOS Apps: Test-Driven Development and Automated Deployment

View more presentations from rudyjahchan

Recently, Jonah and I have been exploring test-driven development and automated deployment on the iOS platform. As we were both attending iOSDevCamp 2011, we decided to give a lightning talk summarizing our discoveries and to generate excitement within others in the community to start their project on the right foot by testing right from the start. While it wasn’t recorded, here is some of the ground we covered in the brief time we had.

tl;dr – Too Long; Didn’t Read »

Download: .ppt | .pdf

Unit Testing

After briefly reviewing why we test first (you are doing that, aren’t you?) we started with how to unit-test the behavior of classes in Objective-C. To Apple’s credit, they’ve bundled a unit-testing framework and the tools to use it in XCode; SenTestingKit. It follows the xUnit style of writing a series of tests with assertions, with an optional setup and teardown that can be executed before and after each test.

SenTestingKit works but is by no means our favorite, having issues with readability, mocking, and running from the command-line. Luckily, there are already a number of great tools to address these issues.

OCHamcrest is the Objective-C port of the popular matching library Hamcrest. Simply including the framework in your project gives you many useful matchers to allow you to write with more readability:

Likewise, adding the OCMock framework provides a wealth of mocking functionality including stubbing of methods, verifying execution of expected methods, partial mocking allowing passthrough to real instances, and even the ability to swap in new implementations for methods.

Finally, the Google Toolbox for Mac and GHUnit are excellent test runners, extending functionality including the capability to run headlessly.

Behavior Driven Development

For a few years now the testing community has been moving towards a more BDD or “spec” style of test-writing, allowing for better structure, more readability, and less repetition of code. The classic example of this is the Ruby based RSpec framework:

For those interested more about writing tests in this style, the RSpec Book is an excellent introduction and reference. While its examples are language specific, its principles can be applied to similar frameworks.

Cedar from Pivotal Labs, takes full advantage of the new Objective-C blocks to mimic RSpec’s structure:

Interestingly, instead of plugging into SenTestingKit, Cedar runs the specs as an app, allowing it to be compiled for the simulator or hardware device. Additionally, Cedar has baked in support to provide colorized output when run from the command-line; look into the accompanying project Rakefile to see how it is done.

A more recent entry into the game is Kiwi. Kiwi is structurally similar to Cedar but provides much cleaner assertions:

Using Kiwi is as simple as including its source in your project. Since it sits atop of SenTestingKit, you can include running all tests as part of your XCode build process. Our brief usage of Kiwi has shown it has some issues running UI components but it does hold promise.

Integration Testing

More recently our explorations have taken us into looking for an integration and acceptance testing framework to include in our process. While we could write test or specs to do end-to-end testing, it’s best to separate the concerns; in fact, we should start by writing integration tests that focus on the user experience!

Again, using our experiences with web development in Ruby, we looked for something similar to Cucumber. Its use of plain text and the tools to parse it are an ideal way to specify what a system is expected to do:

Is there a way to write Cucumber specs for iOS? There is! In fact there are two; iCuke and Frank.

However, there are problems with both projects. iCuke launches the application in the simulator and automatically launches an HTTP interface to interact with the screen. The project also hasn’t seen a commit for over a year, though a number of forks seem to be actively developed. Frank, the other Cucumber based library, is actively maintained and follows a similar strategy of embedding an HTTP server.

A Frank Cucumber driver then communicates with the server to fire-off user events. Our hesitation to use it comes in the amount of setup it requires with care needed to ensure you don’t include it in a release build of the application. And for both iCuke and Frank, the idea of running an entire server feels incredibly heavy for what we want to do. Is there another option?

Strangely enough, a possible candidate was released into the wild the day before our talk. KIF, short for Keep It Functional, from Square aims to minimize the layers and load in order to test like a user. Each step in a scenario focuses on a single user action, targeting an interface component through the accessibility attributes. Well documented, and with the ability to capture screenshots (and video come the release of Lion), KIF has a lot of potential and we look forward to exploring it in the future.

Continuous Integration and Automated Deployment

We close the loop on the full agile process through continuous integration and automated deployment, ensuring code stability and getting it out in the hands of testers. We’ve previously documented how to do so in other posts, including how to build XCode projects and run tests from the command-line, as well as rolling your own Over The Air distribution. Jonah provided a full script that carries out all these steps from start to finish.

Using Jonah’s script and following the lead of Cedar, I started creating a set of Rake tasks to carry out each step, from building to headlessly running specs to signing – all from the command-line (it can be found here). I also decided to use TestFlight to distribute and target builds; it provides a intuitive interface for our pre-release users to access the latest builds and easy to use tools for developers to manage those releases. Even better, their upload API allowed us to write a Rake task to deploy right from the command-line.

What’s Missing

While we’re excited to see all these great tools emerging for iOS development, there is still a LOT to be done to get us anywhere near the ease of writing tests that the Ruby world currently enjoys. While the nested spec-style of code alleviates duplication, you can still end up with large blocks of code that look like this:

This doesn’t really tell us anything about what we are trying to do! It would be far more pleasant to have a library like Factory Girl and simply write:

Similarly, while CoreData provides a way to update schemas from one release to another, there is no proper migration path that can move forward and back to ensure compatibility. Add to that time-based testing like Timecop or automatic testing via Guard and you have a slew of great projects that we could work on.

In Conclusion

As you can see, there are a number of solid options to test and deploy your iOS projects. Personally, I’m looking forward to bringing KIF into my current stack of Cedar, OCHamcrest, and OCMock. More importantly, there is plenty of room for new tools to help these processes. I plan on releasing a gem of the Rake tasks we’ve whipped up, and hope to write an interpreter of gherkin (the language cucumber features are written in) to KIF. What would you like to see?


tl;dr

  • Test-driven principles still apply in iOS development.
  • While out-of-the-box unit testing in XCode comes in SenTestingKit, it is somewhat limited. Use OCHamcrest, OCMock, GTM, and GHUnit to expand functionality
  • Alternatively, for a more BDD approach, use Cedar or the newer Kiwi.
  • Cucumber driven integration tests have been implemented with iCuke and Frank,while the recently released KIF provides for doing the same in Object-C.
  • Command line builds, running tests, and Over The Air distribution are all possible and documented. We’ve done it as a bash script, or rake tasks.
  • TestFlight provides not only a great way for your pre-release users to get your builds, but an API to automate sending it to them.
  • There is much work to be done; factories, fixtures, time manipulation, … why not dive in?

About Rudy Jahchan

Rudy’s fascination with computer programming began at age 10 when he mistakenly picked up the Micro-Adventure book Space Attack; he thought it was going to be about Star Wars. That happy accident led him to graduate from McGill University in Computer Science and start a 12 year career in software development playing with a wide range of technology; everything from web applications to cryptology to NoSQL.
This entry was posted in Mobile, Process. Bookmark the permalink.

10 Responses to Getting “Test”-y in iOS Apps: Test-Driven Development and Automated Deployment

  1. Dan Fabulich says:

    This article was published today, but it’s extremely out of date. If you want to do automated testing of iOS apps, you should be using Xcode’s Automation Instrument, released at WWDC 2010. It’s part of the Instruments app which ships with Xcode, and it allows you to write code in JavaScript to completely automate your UI using iPhone’s accessibility hints. http://developer.apple.com/library/ios/#documentation/DeveloperTools/Reference/UIAutomationRef/_index.html

    Why mess around with half-baked open source tools when Apple ships a finished product?

    • Rudy Jahchan says:

      Not including Javascript AI automation is an oversight on our part and in truth we haven’t really delved that much into it. Thanks for mentioning it, and pointing out on Twitter that we could use Cucumber for JS to drive it. Definitely will look into, especially if we can run it headless!

  2. Jared Carroll says:

    Nice recap Rudy. Glad to see there’s interest in testing in the Objective-C community. When I first got into iPhone development a few years ago it was clear the community didn’t see the benefits of automated testing; instead focusing on time to market and greed. For Rubyists coming over to iOS, Cedar is very familiar but its matchers aren’t that intuitive; Kiwi’s matchers look much cleaner. I’m probably most excited to check out Square’s newly released KIF.

  3. Luke Redpath says:

    Dan: I find it very unfair to dismiss various open-source projects as “half-baked” when Apple’s on UIAutomation framework has it’s own limitations (namely it’s reliance on Instruments making it hard to run from the command line and as part of a CI setup) and has been incredibly lacking in documentation ever since it’s release.

    Rudy: if you guys are using TestFlight and Rake, you might find my gem of some use: https://github.com/lukeredpath/betabuilder

  4. Arjun says:

    Would be nice to have a +1 button to help us Google+ers.

  5. Great post. That’s a really nice summary of the tools out there. On most projects up until now, I’ve used a combination of GHUnit and Frank. For any new projects, I’m keen to try Kiwi and KIF. I’m pretty sure Frank is heading in the direction of using KIF as the native driver as well, so it might be worth combining efforts if you’re still attempting that.

    In regard to the dangers of Frank being included in your production target, I don’t see how having a separate ‘Frank’ target for testing that includes the server is any different to having a separate ‘KIF’ target. UIAutomation manages to drive the app through Instruments, so if we could reverse engineer how that works that could solve the problem of having a server.

    I attempted a smaller comparison, just Frank/KIF/UIAutomation which might be relevant to people reading this post http://sgleadow.github.com/blog/2011/10/26/which-automated-ios-testing-tool-to-use/

  6. Ajit Gosavi says:

    This is great post..really helpful..do you have any ideas of how to control installations on device and remotely triggering instruments automation scripts. we have automation scrips in test app and would like to deploy test app and application under test.

  7. Tim says:

    This was a great post and link to tools….I am especially interested in how people use Cedar to get Rspec type BDD into IOS development.

  8. Pingback: iOS Testing | CodePlease

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>