Running Xcode 4 unit tests from the command line

Posted on by in Mobile

Command line builds for Xcode 4 projects are a good first step but I really want to get my project’s tests running on a continuous integration server again. Since “test” isn’t a valid build action to pass to xcodebuild I’ve been looking for a configuration which would allow me to run tests in a headless environment.

I expect that GTM, GHUnit, and Cedar will all have reliable support for Xcode 4 projects eventually but I would like to start with just seeing some SenTestingKit tests pass.

I have been able to run test suites from the command line but only for “logic” tests. From the iOS Development Guide:

Xcode offers two types of unit tests: logic tests and application tests.

Logic tests.
These tests check the correct functionality of your code in a clean-room environment; that is, your code is not run inside an application. Logic tests let you put together very specific test cases to exercise your code at a very granular level (a single method in class) or as part of a workflow (several methods in one or more classes). You can use logic tests to perform stress-testing of your code to ensure that it behaves correctly in extreme situations that are unlikely in a running application. These tests help you produce robust code that works correctly when used in ways that you did not anticipate. Logic tests are iOS Simulator SDK–based; however, the application is not run in iOS Simulator: The code being tested is run during the corresponding target’s build phase.

Application tests.
These tests check the functionality of your code in a running application. You can use application tests to ensure that the connections of your user-interface controls (outlets and actions) remain in place, and that your controls and controller objects work correctly with your object model as you work on your application. Because application tests run only on a device, you can also use these tests to perform hardware testing, such as getting the location of the device.

I am able to create a set of logic tests as a separate build target and scheme which will successfully run as part of an xcodebuild build.

  1. Create a new “LogicTests” unit test build target.
  2. Leave the “Test Host” build setting blank.
  3. Set the “Test After Build” build setting to “No”.
  4. Do not write tests which require an application to be present (ie no window or application delegate available).
  5. Create a new “LogicTests” scheme which includes the “LogicTests” build target in its run action.
  6. Run the “LogicTests” scheme using xcodebuild.
Scheme for running logic tests

Scheme for running logic tests

> /Developer_Xcode4/usr/bin/xcodebuild -workspace WrappingScrollView.xcworkspace -scheme logictests -sdk iphonesimulator4.3 -configuration Debug clean build

Run test suite logictests
Test Suite ‘logictests’ started at 2011-04-06 07:44:57 +0000
Run test case testExample
Test Case ‘-[logictests testExample]’ started.
/Users/Jonah/Desktop/WrappingScrollView/WrappingScrollViewDemo/logictests/logictests.m:30: error: -[logictests testExample] : Unit tests are not implemented yet in logictests
Test Case ‘-[logictests testExample]’ failed (0.000 seconds).

Test Suite ‘logictests’ finished at 2011-04-06 07:44:57 +0000.
Executed 1 test, with 1 failure (0 unexpected) in 0.000 (0.000) seconds

That’s not a great solution but it at least allows me to run some tests on a continuous integration server. Hopefully I can find a way to run application tests through xcodebuild as well.

In case it is of any help to other developers struggling with the difference between command line and Xcode builds. The “Run Script” build phase of a unit test build target just runs “${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests”. The RunUnitTests script in turn invokes a platform specific script like “${SYSTEM_DEVELOPER_DIR}/Platforms/iPhoneSimulator.platform/Developer/Tools/RunPlatformUnitTests” which ultimately calls RunTestsForBundle in “${SYSTEM_DEVELOPER_DIR}/Tools/RunPlatformUnitTests.include”. Hopefully it will prove possible to use these scripts to match the behavior seen when running tests from within Xcode to automatically run application tests.


Feedback

  Comments: 21


  1. Any progress getting application tests to work?


  2. Don’t know that this will satisfy everybody’s needs but, if all you want is a command-line window in Xcode 4, you can start up Xcode 4 and drag a command-line .c file to the Xcode icon in the dock. It will open in a command line window that is neat and less cluttered than the comparable window in Xcode 3.


  3. I would just be happy to run logic tests in xcode4 without the simulator launching. I understand it needs to launch for application tests, but I don’t need any iOS classes for my logic tests. Any ideas on how I could accomplish this?


  4. The above xcodebuild command just builds my LogicTests scheme. It is not running any testcases! Any settings I need to change?


  5. I can’t get this to work. The tests run fine in Xcode, but on the command line I get the following error:

    /Developer/Tools/RunPlatformUnitTests.include:419: note: Running tests for architecture ‘i386′ (GC OFF)
    2011-10-18 18:15:47.373 otest[53985:7803] The test bundle at /Users/ben/Library/Developer/Xcode/DerivedData/Deli-gspotoqkkxdkiqddximbbiwaivvh/Build/Products/Debug-iphonesimulator/deliappTests.octest could not be loaded because a link error occurred. It is likely that dyld cannot locate a framework framework or library that the the test bundle was linked against, possibly because the framework or library had an incorrect install path at link time.
    /Developer/Tools/RunPlatformUnitTests.include:448: error: Failed tests for architecture ‘i386′ (GC OFF)

    I’m not sure what the error is, any ideas?


  6. The article is very interesting, but how can I simulate, for example, 20K users running my mobile app at the same time?


    • Well you wouldn’t simulate it on the client or in the simulator since you’d never have 20000 people using your app on one phone at the same time and the simulator instance represents one device.

      You’ll be hitting the server with requests that would be same requests that come from your app. And you can do that from 20k threads for example – one thread per user. You can use any load testing tool to do that – jMeter should do the trick.


  7. I’ve seen a replay to this here:
    http://longweekendmobile.com/2011/04/17/xcode4-running-application-tests-from-the-command-line-in-ios/

    what I still don’t understand is how the application is actually _run_ on the device. What are the arguments for ${SYSTEM_DEVELOPER_DIR}/Tools/RunUnitTests?


  8. Thank you so much for this article. It helped me a lot to get my own CI setup up and running. I also wrote an article about getting application tests in general and Kiwi specs in particular to run properly from the command line in a Jenkins environment: Continuous Integration of iOS Projects using Jenkins, CocoaPods, and Kiwi. Maybe this is of help to the readers of this article as well.


  9. My build of the logic test scheme is succeeded but any test is run. Could u give me light?

Your feedback