- (void) testViewBinding {
	TestableSimpleViewController *viewController = [[TestableSimpleViewController alloc] initWithNibName:@"TestableSimpleViewController" bundle:nil];
	[viewController loadView]; //It is not strictly necessary to call loadView for this test as we access the view property which will call loadView if view is nil
	assertThat(viewController.view, isNot(nilValue()));
	assertThat(viewController.button, isNot(nilValue()));
}
 
- (void) testUIButtonActionBinding {
	TestableSimpleViewController *viewController = [[TestableSimpleViewController alloc] initWithNibName:@"TestableSimpleViewController" bundle:nil];
	[viewController loadView]; //Here we must call loadView since we have not accessed the controller's view property to trigger view loading from the nib
	assertThat([viewController.button actionsForTarget:viewController forControlEvent:UIControlEventTouchUpInside], onlyContains(@"testAction", nil));
}

Testing View Reloading
View controllers should be able to unload their views if the application receives a memory warning while the view is not visible and then reload the view when it is needed again. We can reproduce that same sequence of messages in a test. For a non-trivial controller we would add additional assertions to test that any other view dependent properties were correctly released and recreated as the view was unloaded and reloaded.

- (void) testViewUnloading {
	TestableSimpleViewController *viewController = [[TestableSimpleViewController alloc] initWithNibName:@"TestableSimpleViewController" bundle:nil];
	[viewController loadView];
	assertThat(viewController.view, isNot(nilValue()));
	[viewController didReceiveMemoryWarning];
	assertThat(viewController.button, is(nilValue())); //Note that while viewController.view is nil here we cannot test that directly as accessing view will trigger a call to loadView. Instead we can only test that our outlets have been released as expected.
	[viewController loadView];
	assertThat(viewController.view, isNot(nilValue()));
}

Testing Switching Between View Controllers
View controllers frequently have dependencies on other view controllers. If those dependencies are tightly coupled it becomes very difficult to test the behavior of a single view controller. Instead we can expose those dependencies and replace them with mock objects in order to isolate the controller we are currently testing. Here we have a simple table view controller which presents a detail view controller when a table cell is selected. By mocking both the navigation controller used to present that detail view and the detail view itself we can capture and verify the table view controller’s behavior.

- (void) testSelectingACellPushesADetailView {
	id detailViewController = [OCMockObject niceMockForClass:[GenericDetailViewController class]];
	NSDictionary *data = [[NSDictionary alloc] init];
	UINavigationController *navController = [[UINavigationController alloc] init];
	id navigationController = [OCMockObject partialMockForObject:navController];
 
	TestableTableViewController *viewController = [[TestableTableViewController alloc] initWithNibName:@"TestableTableViewController" bundle:nil];
	viewController.detailViewController = detailViewController;
	viewController.modelData = data;
 
	[navigationController pushViewController:viewController animated:NO];
	assertThat(viewController.navigationController, isNot(nilValue()));
 
	[viewController loadView];
	[[detailViewController expect] setModelData:data]; //expect that the detail view will be given some data to display
	[[navigationController expect] pushViewController:detailViewController animated:YES]; //expect that the detail view will be pushed onto the nave controller
	[viewController tableView:viewController.tableView didSelectRowAtIndexPath:[NSIndexPath indexPathForRow:0 inSection:0]];
	[detailViewController verify];
	[navigationController verify];
}

So what makes test-driven controllers different?
I certainly could have written these example controllers without any tests around them. However by writing tests I was encouraged to make a couple of changes to the way I structure my code.

• My controllers expose dependencies that might otherwise remain buried in their implementations. In the example above I exposed the detail view controller my controller used so that I could replace it with a mock object. That made it possible to test that the expected data was passed to the detail view but it will also make it easy to switch to a different detail view controller in the future.
• By focusing on testing just one class at a time I am encouraged to create well defined interfaces and a clear set of responsibilities for each class. That in turn leads to more modular code which is easier to maintain and reuse.
• I write bugs but if I write a test to reproduce them and then prove that they are fixed I shouldn’t ever see the same bug twice. It is much easier to catch a missing nib binding because a test failed than it is to see the app crash when displaying some view and have to track down that exc_bad_access error in the debugger.
• Most importantly by writing tests I can have confidence that my code works as I expected, not only when I first write it but also every time I run my tests (which happens every time I build the app). With that feedback I can change one controller without fearing that I might have broken something somewhere else in the app.

For those of you who caught it last week at the SDForum, here are the links people requested:

• Maven plugin for Javascript Unit Testing with Rhino/ScrewUnit (from C5)
• The current code for the “wizardize” example we walked through.
• Blue Ridge (Rails integration tool)
• ScrewUnit
• JResig’s Presentation on JS Testing
• The slides we used

To sign up for next Wednesday, go to the EBig site.

The teaser for it… Recent evolutions in JavaScript testing frameworks now allow creating test suites, test-driving development, and running tests on a continuous integration server. This allows us to support more complex JavaScript, have confidence in the implementation, and push more of the logic from the server into the browser, reducing the load on the server…

Hope to see you there.

<plugin>
    <executions>
        <execution>
            <goals><goal>javascript-test</goal></goals>
        </execution>
    </executions>
    <groupId>com.carbonfive.javascript-test</groupId>
    <artifactId>javascript-test-maven-plugin</artifactId>
    <version>1.0-beta1</version>
    <configuration>
        <includes>
            <include>src/test/javascript/suite*.html</include>
        </includes>
    </configuration>
</plugin>

You will probably also need to add Carbon Five’s public repository so that maven will know where to grab the plugin:

<pluginRepositories>
    <pluginRepository>
        <id>c5-public-repository</id>
        <name>Carbon Five Public Repository</name>
        <url>http://mvn.carbonfive.com/public</url>
    </pluginRepository>
</pluginRepositories>

The <configuration> section contains standard maven <include> and <exclude> tags to allow you to specify any Screw.Unit test suite html files you would like to include in your build.

Now when maven executes its test phase, it will include your javascript tests as well. Any test that fails will halt the build. When the test phase is complete the target directory will contain a ‘screw-unit’ directory that contains the test reports. There will be two files for every file included in the test execution The first is an XML file named TEST-dir1.dir2.dir3.filename.html.xml that contains the test results for that file in a reasonable approximation of JUnit’s xml format. The other file will be named dir1.dir2.dir3.filename.html and will contain a sort of snapshot of the test suite’s html at the time the tests completed (the javascript will be removed and the css will be baked in).

Like Blue Ridge, the javascript-test-maven-plugin relies on the excellent env.js project and the less-excellent Rhino project to emulate a browser in Java. Unfortunately this combination creates the possibility of discrepancies between the browser and build results, and the probability of relatively slow test execution. To keep the simulation as realistic as possible, javascript-test-maven-plugin recreates the test environment for each Screw.Unit html file, so the more tests you can cram into a single file the faster your builds will run.

There are a few directions for future work. I’d like for the plugin to handle other browser-based javascript testing frameworks. Screw.Unit is popular, but seems suboptimal in many respects. The only limiting factor here is the plugin’s ability to interpret the results. If env.js could run in javascript interpreters other than Rhino, I’d like to give the the plugin the capability to fork out to spidermonkey for better compatibility and performance. Modifying the plugin would be relatively easy, but env.js appears to have some significant dependencies on Java.

Check it out if you have a need for this kind of thing and let us know if you have any problems or suggestions.

There are a couple of different options for JS testing available at this point but Blue Ridge’s plugin is the first I have seen which made writing test driven code as easy as I have come to expect while still providing all the functionality I need. In particular;

• Running my tests in a browser is fast. Refreshing fixture pages to rerun tests is dramatically more efficient than waiting for selenium tests to cycle.
• I can debug my tests as they run against their fixtures pages and inspect or manipulate the DOM at any point.
• I have mock and fake objects available to test behavior and stub out dependencies.
• My JS tests run in continuous integration, just like all the other tests.

By writing fixtures pages and tests first I am immediately using all of my code in two locations and running it against slightly different DOMs. That has pushed me to encapsulate most of my code as jQuery plugins and think carefully about how much I depend on the page’s structure. I end up with terse selectors instead of using whatever attributes are readily available and a clear configuration point for my page’s behavior as I chain together plugin calls in $(document).ready().

We’d like to see what other developers have done and share what we have learned so far so Carbon Five is hosting an open discussion of JavaScript testing in our office this week.

Topic: Javascript Testing
When: Tuesday Nov 17
6pm: arrive/network
6:30 – 8 presentation and discussion
Where: Carbon Five office: 171 2nd Street, 4th Floor

Please RSVP if you would like to attend.

Here’s the high-level flow that the functional test runner provides:

1. Load fixture data from a DBUnit dataset.
2. Download and install the application server (if necessary).
3. Start the application server (using Cargo).
4. Deploy the application, waiting until it’s completely started.
5. Run one or more functional tests (using your preferred testing framework – Selenium, HTMLUnit, etc)…
   If a test dirties the database in a manner that must be reset, the test class can be marked with the @DirtiesDatabase annotation. This will reload the database fixture and optionally restart the application.
6. Shutdown the application server.

It’s easy to use… First add the maven dependency:

  ...
  <repositories>
      <repository>
          <id>c5-public-repository</id>
          <name>Carbon Five Public Repository</name>
          <url>http://mvn.carbonfive.com/public</url>
       </repository>
  </repositories>
  ...
  <dependency>
      <groupId>com.carbonfive.test-support</groupId>
      <artifactId>test-support</artifactId>
      <version>0.9.2-m1</version>
      <scope>test</scope>
  <dependency>
  ...

And then write your first functional test:

@RunWith(FunctionalTestRunner.class)
public class FunctionalTestRunnerTest
{
    // This field is automatically injected by the test runner and references
    // the functional test properties (see below).
    private Properties properties;
 
    @Test
    public void propertiesShouldBeSetByRunner()
    {
        assertThat(properties, not(nullValue()));
        assertThat(properties.size(), greaterThan(5));
    }
 
    @Test
    public void applicationShouldBeDeployed() throws Exception
    {
        // Test using your preferred framework... just using URL as an example.
        URL root = new URL(format("http://localhost:%s/", properties.getProperty("appserver.port")));
        String content = IOUtils.toString(root.openStream());
        assertThat(content, containsString("Orange"));
    }
}

Lastly, create the configuration file that goes along with it. By default, it’s called functional-test.properties and lives at the root of the classpath (src/test/resources/ for maven users). Here’s an example:

# Application server
appserver.container         = tomcat6x
appserver.installer         = http://www.apache.org/dist/tomcat/tomcat-6/v6.0.20/bin/apache-tomcat-6.0.20.zip
appserver.port              = 8080
appserver.logging           = false
 
# Application configuration
app.war                     = ./gearlist-webapp/target/gearlist-webapp-1.0-SNAPSHOT.war
app.context                 = ROOT
 
# Database fixtures
fixture.restart_application = false
fixture.file                = classpath:/database-fixture.xml
 
 
# Application configuration (not used by the functional test runner)
db.driver                   = com.mysql.jdbc.Driver
db.host                     = localhost
db.name                     = gearlist_func_test
db.url                      = jdbc:mysql://${db.host}/${db.name}
db.username                 = root
db.password                 =

Property values can reference other properties (e.g. db.url) and they’re all set as system properties so it’s easy to override your application properties for functional testing.

Both the database fixture loading and the application server steps can be customized by plugging in a new DatabaseFixtureLoader or ApplicationServerManager, respectively.

Since it’s a JUnit extension, functional tests run exactly the same whether they’re triggered from the IDE or the command line. This approach is a response to how clunky it felt using Maven to launch the application server before running the tests. Also, with only one master orchestrator, it’s easier to handle redeploying the application if necessary.

Want to see a real example? I’ve written a sample application called “gearlist” that’s available via public svn (press cancel if asked for authentication credentials). It includes several HtmlUnit and Selenium functional tests, as well as other types of tests using some of the other test utils I’ve written. Look for the readme for details.

Carbon Five Test Support is hosted on google code, where you can read what documentation there is and peruse the source.

There are a few things to work on, including better exception handling during initialization and some flexibility around the Cargo configuration, however, it’s pretty stable and there’s no reason not to use it now. Give it it try and give me your feedback. The plan is to tighten it up and write some documentation for the 0.9.2 release.

I’ll be down in Palo Alto speaking about Automated Web Application Testing Tuesday October 6th. If you’re interest in getting a peek at the typical Carbon Five Java web architecture along with a variety of strategies and techniques for testing, c’mon down and join us. The session will be primarily code-driven. I’ll implement a few features along with unit, integration and functional tests and show off some of the techniques and custom tools that help keep things simple and easy during development. Some of the topics include:

• Brief overview of our typical web architecture and tools stack
• Differences between Unit, Integration and Functional Tests
• Dealing with the database (schema and fixtures)
• Where does test coverage pay off the most?
• In-browser and out-of-browser functional testing
• Carbon Five best practices, custom tools and techniques
• And more…

You can find the gritty details on the SDForum site.

Doors at 6:30 and the show begins at 7:00. Hope to see you there!

Here are the artifacts from the presentation:

• Presentation (.pdf)
• Source Code (svn)
• Build and Run Instructions (readme.txt)

To reuse my templates checkout a copy from http://svn.carbonfive.com/public/jonah/Xcode-templates/Xcode/ to “/Library/Application Support/Developer/Shared/Xcode/” and restart Xcode.

Installing the template files:

I installed Apple’s developer tools using the default path so my copy of Apple’s Xcode templates is located in “/Developer/Library/Xcode/File Templates” and “/Developer/Library/Xcode/Project Templates”. Installing the iPhone SDK added additional templates in “/Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/”. Those were useful examples to base my own templates on but I needed to save my new templates separate from Apple’s default configuration files.

The Xcode release notes state that custom support files including project and file templates belong under “/Library/Application Support/Developer/” or “~/Library/Application Support/Developer/”. I didn’t need to restrict the visibility of these templates to a specific version of Xcode and I wanted them to be available to any user account so I put them in “/Library/Application Support/Developer/Shared/Xcode/Project Templates” and “/Library/Application Support/Developer/Shared/Xcode/File Templates”.

Building a template project:

The Problem

On our work on gobalto.com, we spend time to have good fixture data for our tests– data that can represent all the important application states that our tests require. As a result, our tests are very dependent on the data. It’s important that someone doesn’t inadvertantly change it in subtle ways. This has led us to write not only asserts at the end of tests, but pre-conditions as well. For example,

    ...
    inotech = companies(:inotech)
    assert inotech.services.public.include?(categories(:a))
    assert inotech.services.public.include?(categories(:b))
    assert inotech.services.public.include?(categories(:c))
 
    post :edit_services_dialog, :id=>inotech.id,
                                          :service_category_id=>categories(:a).id
    inotech.reload
 
    assert inotech.services.public.include?(categories(:a))
    assert !inotech.services.public.include?(categories(:b))
    assert !inotech.services.public.include?(categories(:c))

Although the pre-conditions were introduced to guard against accidentally changing fixture data (or just to figure out what’s going on), we don’t necessarily delete them. They provide stronger tests. And in some ways, leaving in pre-conditions make the code more readable by providing documentation of your assumptions to the readers. The code above would be hard to follow without the clarifying pre-conditions. What do we expect to change and what stays the same?

Unfortunately all these asserts make the tests twice as long. And there’s a subtle readability problem: there’s no relationship between the corresponding pre- and post-conditions. In the example above, you have to scan carefully to see that the three assertions are repeated, but negated (b and c only). The reader must mentally put pieces together. And it’s not DRY. Using local variables doesn’t help much.

The Solution

I was inspired by a nice little test helper called assert_difference. It takes a string to evaluate and a block to execute. It’s useful for checking on state changes– especially database changes– during a test:

assert_difference('Company.count', -1) do
    Company.delete_one
end

(Without this method, we rely on a count of all database records, or merely look for specific ones. The former approach leads to brittle tests, and the latter to incomplete assertions.)

A limitation of assert_difference is that it only deals with integers. What if it were generalized? Here goes:

    i = true
    assert_changes 'i' => false do   # read as: i changes to false
      i = false
    end

The string passed to assert changes is evaluated in the block context, both before and after the block is run. So this block asserts that i becomes false (and by deduction, starts out as true). It executes asserts on both ends, just like we want.

Of course sometimes you want to be explicit about a state change, so you can specify both the starting and ending values using an array:

    o.answer = 'yes'
    assert_changes 'o.answer' => ['yes','no'] do
      o.answer = 'no'
    end

To handle the original example, you can pass multiple pre/post conditions of arbitrary complexity and they are all evaluated before and after the block is executed:

      assert_changes 'post(:a).status' => [:preview, :published],
                            'comment(:c).status' => [:preview, :deleted] do
        ...
      end

Finally, I added support for a :no_change symbol. Now I can re-write my original problem in a clearer form:

    assert_changes
        'inotech.services.public.include?(categories(:a))' => [true, :no_change],
            'inotech.services.public.include?(categories(:b))'=>false,
            'inotech.services.public.include?(categories(:c))'=>false do
      post :edit_services_dialog, :id=>inotech.id, :service_category_id=>categories(:a).id
      inotech.reload
    end

For completeness, I added assert_no_changes, with slightly extended parameter possibilities:

    i,j = 'hello','hi'
    assert_no_changes 'i' do ...  # i (before) == i (after)
    assert_no_changes 'i'=>'hello' do ... # i == 'hello' before and after
    assert_no_changes ['i','j'] do ... # neither i or j change
    assert_no_changes 'i'=>'hello','j'=>'hi' do # or be explicit

Update: This code and documentation is now available on github: http://github.com/ndp/assert_changes/tree/master

Thoughts? Comments?

I recently came across a scenario where I needed to make an assertion on an argument passed to an expected method invocation but OCMock did not provide the constraint I needed. I was testing a FileDetailsViewController that pushes a new FileContentsViewController on to the view stack when a button is pressed. I mocked the UINavigationController so that I could add an assertion that pushViewController:animated: is called with the right arguments.

The meat of my test is:

@interface FileDetailsViewControllerTest : SenTestCase {}
    FileDetailsViewController *controller;
    id navigationController;
@end
 
@implementation FileDetailsViewControllerTest
 
-(void) setUp {
    controller = [[FileDetailsViewController alloc] initWithNibName:@"FileDetailsView" 
                                                             bundle:nil];
    navigationController = [OCMockObject mockForClass:[UINavigationController class]];
    controller.navigationController = navigationController;
}
 
-(void) testShowContentLoadsView {
    [[navigationController expect] 
         pushViewController:[OCMConstraint isKindOfClass:[FileContentsViewController class]]
                   animated:YES];
    [controller showContentsButtonWasPressed];
}
 
-(void) tearDown {
    [navigationController verify];
}
 
@end

Note that in this example, I some fanciness is required to assign controller.navigationController = navigationController since it is a readonly property in the UIViewController API. I’ll explain what I did in a later post.

The OCMock feature I was missing is the OCMock.isKindOfClass constraint. So I wrote my own.

Following is the header file that declares the isKindOfClass assertion as a static method in an Objective-C category for OCMock. Categories are kind of like Ruby mixins for Objective-C. You can add static or instance methods to an existing class definition. With this approach I can follow the OCMock pattern of providing static factory methods on the OCMock class for the range of constraints available.

//File: OCMockConstraint+Extensions.h
#import <OCMock/OCMConstraint.h>
 
@interface OCMConstraint (Extensions)
 
+ (id)isKindOfClass:(id)value;
 
@end

Following is the implementation, both of my new constraint and the factory method to create it.

//File: OCMockConstraint+Extensions.m
#import "OCMockConstraint+Extensions.h"
 
@interface OCMKindOfClassConstraint : OCMConstraint
{
    @public
    id testClass;
}
@end
 
@implementation OCMKindOfClassConstraint
 
- (BOOL)evaluate:(id)value
{
	return [value isKindOfClass:testClass];
}
 
@end
 
// Static factory method
@implementation OCMConstraint (Extensions)
 
+ (id)isKindOfClass:(id)value {
	OCMKindOfClassConstraint *constraint = [OCMKindOfClassConstraint constraint];
	constraint->testClass = value;
	return constraint;
}
 
@end

This implementation shows both how to write a custom constraint for OCMock and a use of Objective-C categories to provide a nice API for using it.

In many cases, you’ll want to write a custom constraint that is specific to your test scenario and not appropriate to be shared globally in this manner. In this case you’ll probably follow a similar strategy but implemented in your test class instead of mixed in to OCMockConstraint.

I’ve been bootstrapping with the beta version of the Pragmatic iPhone SDK Development book but was disappointed to find no discussion of unit testing for the iPhone. So I’ve been piecing it together as I go.

GTM

Unit testing in Objective-C is provided by the SenTestingKit framework that installs with XCode in /Developer/Library/Frameworks/. However it is not compatible with the iPhone SDK. The Google Toolbox for Mac provides an implementation of SenTestCase for the iPhone that I am using instead. The instructions provided on that page worked for me with one exception that I included as a comment on that page:

I followed the instructions above but was unable to see any messages in the console. I did not see the behavior described above as:

Your target should now build cleanly, and if you check the build log you should see something like: “Executed 0 tests, with 0 failures (0 unexpected) in 0.001 (0.001) seconds” at the end.

It turned out that I needed to set the base SDK in the inspector for my target to an iPhone simulator even though the Overview showed I was building for simulator debugging.

I created a ‘Unit Tests’ target per the GTM directions to run my tests. Now I am happily running my tests with command-B. One point that could be a problem when getting into continuous integration is that my Unit Tests target reports ‘Build succeeded (1 error)’ when a unit test fails instead of having the build actually fail.

I have not gotten into the “Advanced Stuff” described on that page.

Hamcrest

The SenTestingKit assertion methods are pretty lame. They’re verbose and require that you provide a string message for your expectation (e.g. STAssertEqualStrings(a1, a2, description, …)). We’re using and like Hamcrest on many of our Java projects. I was psyched to see there is an Objective-C implementation.

There is no download available for Hamcrest Objective-C so I checked it out from SVN and built the framework product in Xcode for OS X 10.5/i386.

I struggled for a while to get the dependency on the hamcrest.framework build product into my project. I ended up with this solution:

1. Create a ‘Frameworks’ folder in my project root
2. Copy hamcrest.framework to Frameworks/
3. In Xcode, right click on ‘Frameworks’ and ‘Add existing framework…”
4. Browse to the hamcrest.framework folder and add it, making sure to add it to my Unit Tests target only
5. Right click on the Unit Tests target to add a new Copy Files Build Phase that copies to the Products directory and put that build phase early in the target steps
6. Add hamcrest.framework to that build phase by dragging from Frameworks
7. Include Hamcrest in your test class and you’re good to go

#import "GTMSenTestCase.h"
#define HC_SHORTHAND
#import <hamcrest/hamcrest.h>
 
@interface ExampleWithAssertThat : SenTestCase
@end
 
@implementation ExampleWithAssertThat
 
- (void) testUsingAssertThat
{
    assertThat(@"xx", is(@"xx"));
    assertThat(@"yy", isNot(@"xx"));
    assertThat(@"i like cheese", containsString(@"cheese"));
}
 
@end

OCMock

iPhone development requires a lot of mucking around with UI framework classes like UITableView, UIApplicationDelegate and so on. These classes have a lot of dependencies and are depended upon all over the place which can make it hard to get them under test.

Mock objects are a great solution to this problem and OCMock provides an implementation for Objective-C. In this case, the OCMock.framework provided in the OCMock binary release worked for me when installed in the same manner as described for Hamcrest above.

This example of testing [UIApplicationDelegate applicationDidFinishLaunching] illustrates its use pretty well. I have a simple test that creates a mock UITableView to asserts a cell is correctly created in [UITableViewController tableView:cellForRowAtIndexPath:]. It looks like:

#import "GTMSenTestCase.h"
#define HC_SHORTHAND
#import <hamcrest/hamcrest.h>
#import <OCMock/OCMock.h>
 
#import "SortedStateViewController.h"
#import "Country.h"
 
@interface SortedStateViewControllerTest : SenTestCase {}
 
SortedStateViewController *controller;
 
@end
 
@implementation SortedStateViewControllerTest
 
- (void) setUp {
    controller = [[SortedStateViewController alloc] init];
    Country *country = [[Country alloc] init];
    [country addState:@"New York" withPopulation:19306183 andArea:47213];
    [country addState:@"California" withPopulation:36457549     andArea:155959];
    controller.country = country;
    controller.key = @"population";
}
 
- (void) testPopulatesCell {
    id tableView = [OCMockObject mockForClass:[UITableView class]];
    [[tableView stub] dequeueReusableCellWithIdentifier:[OCMConstraint any]];
    NSIndexPath *indexPath = [NSIndexPath indexPathForRow:(NSUInteger)0 
                                                inSection:(NSUInteger)0];
 
    UITableViewCell *cell = [controller tableView:tableView 
                            cellForRowAtIndexPath:indexPath];
    assertThat(@"California", equalTo(cell.text));
}
 
@end

Continuous Integration

So I’m pretty happy with where this is now. I’m working through the Pragmatic book TDD as I go. It’s also have an ObjectiveCTest that I add assertions to as I play with features of the language.

Missing from this toolkit is a good continuous integration solution.

There is now a recently released Xcode builder for CruiseControl. However, we use Bamboo for CI for our Java and Ruby projects and are considering moving to TeamCity. It looks like there is an Xcode plugin for TeamCity and some attempts with Bamboo.

More on that later.

We’ve been using DB Unit on our Java projects for years and the mechanics of how it’s used has evolved over time. I’ve recently spent some time making it work a little nicer for how we typically write database tests. What I’ve created makes using DBUnit on a project that is already using Spring and the testing support added in Spring 2.5 just a little easier through the application of convention and annotations.

In general, we’ve adopted the convention of loading data off the classpath from a flat dataset file named after the test located next to the test on the classpath. For example (in the maven standard directory structure):

• src/test/java/com/acme/TripRepositoryTest.java – Java Test Code
• src/test/resources/com/acme/TripRepositoryTest.xml – DB Unit Data Set for TripRepositoryTest

For most tests, the data set is loaded inside the test’s transaction and rolled back when the test completes so that nothing needs to be cleaned up (see Spring’s reference). For other tests — service or integration tests — the data is loaded outside of a transaction and must be cleared out manually. Most projects have a mix of both strategies and both should be easily supported.

When Spring 2.5 came out with its new testing framework, I threw together a custom TestExecutionListener that looks for test methods that are annotated with @DataSet, and when found, loads the data using DB Unit. Here’s a transaction-per-test example:

TripRepositoryImplTest.java – Example transaction-per-test Test Case

@ContextConfiguration(locations = {"classpath:applicationContext.xml"})
public class TripRepositoryImplTest extends AbstractTransactionalDataSetTestCase {
    @Autowired TripRepository repository;
 
    @Test
    @DataSet
    public void forIdShouldFindTrip() throws Exception {
        Trip trip = repository.forId(2);
        assertThat(trip, not(nullValue()));
    }
}

The high-level execution path for this example looks like:

1. Inject dependencies (DependencyInjectionTestExecutionListener)
2. Start transaction (TransactionalTestExecutionListener)
3. Load dbunit data set from TripRepositoryImplTest.xml (DataSetTestExecutionListener) using the setup operation (default is CLEAN_INSERT)
4. Execute test
5. Optionally cleanup dbunit data using the tear down operation (default is NONE)
6. Rollback transaction (TransactionalTestExecutionListener)

Here’s the trimmed down log output for this test:

INFO: Began transaction (1): transaction manager; rollback [true] (TransactionalTestExecutionListener.java:259)
INFO: Loading dataset from location 'classpath:/eg/domain/TripRepositoryImplTest.xml' using operation 'CLEAN_INSERT'. (DataSetTestExecutionListener.java:152)
INFO: Tearing down dataset using operation 'NONE', leaving database connection open. (DataSetTestExecutionListener.java:67)
INFO: Rolled back transaction after test execution for test context (TransactionalTestExecutionListener.java:279)

For this to work in its current incarnation, a single datasource must be available for lookup in the application context. One of the interesting details is what to do with the connection used to load the data. The framework assumes that if it’s a transactional connection it should be left open because whatever started the transaction should do the closing. When it’s non-transactional it’s closed after the dataset is loaded. This convention works well for how I typically write my database tests.

In addition to the @DataSet annotation, we must add the DataSetTestExecutionListener to the set of listeners that are applied to the test class. As in the above example, you can extend AbstractTransactionalDataSetTestCase which does this for you or you can specify the listener using the class-level annotation @TestExecutionListeners (see example). It’s important that the listener is triggered after the TransactionalTestExecutionListener.

If all test methods use the dataset, then the test class (or super class) can be annotated and every test will load the dataset. Also, if a different dataset should be loaded, the name of the resource can be specified in the annotation (e.g. @DataSet(“TripRepositoryImplTest-foo.xml”) or @DataSet(“classpath:/db/trips.xml”)). Lastly, the setup and teardown database operations can be overriden (e.g. @DataSet(setupOperation = “INSERT”, teardownOperation=”DELETE”)).

This functionality is part of the C5 Test Support package and is available in our maven repository. To use it, first add the C5 Public Maven repository to your pom.xml, and then add the necessary dependencies:

pom.xml

<repositories>
    <repository>
        <id>c5-public-repository</id>
        <url>http://mvn.carbonfive.com/public</url>
        <snapshots>
            <updatePolicy>always</updatePolicy>
        </snapshots>
    </repository>
</repositories>
...
<dependencies>
    <dependency>
        <groupId>org.dbunit</groupId>
        <artifactId>dbunit</artifactId>
        <version>2.2.3</version>
        <scope>test</scope>
    </dependency>
 
    <dependency>
        <groupId>com.carbonfive</groupId>
        <artifactId>test-support</artifactId>
        <version>0.6</version>
        <scope>test</scope>
    </dependency>
    ...
</dependencies>

Check out the sample application for details. It’s mavenized and utilizes an in-memory database. Just check it out of subversion, look over the code, and give it a run using your IDE or from the command-line (mvn install). I’d be psyched to hear what you think and of course, welcome comments and suggestions.

Resources:

• C5 Test Support Source: http://svn.carbonfive.com/public/carbonfive/test-support/trunk
• C5 Test Support Maven Home: http://mvn.carbonfive.com/public/com/carbonfive/test-support/0.6/
• Sample Application: http://svn.carbonfive.com/public/christian/spring-dbunit-test-extension/trunk

On my current project, we have a queue of articles that need human-user attention (i.e. editorial moderation). Each article must be doled out to only one moderator and there are multiple instances of the web application servicing requests in the cluster. Imagine tens of thousands of articles per day and a team of moderators churning through them. We can’t rely on Java synchronization because it only works within the JVM instance, not across server instances.

The simplified version of the service interface we’re working on looks like this:

ArticleService.java – Example service interface

public interface ArticleService
{
    Article findNextArticleForModeration();
}

What makes this interesting is that we must ensure that the service doesn’t hand out the same Article to more than one user. This is impossible to assert using a single thread. We’ve all been told that multiple threads and automated testing don’t mix. It’s generally true and should be avoided if at all possible, but in some cases it’s the only way we can truly assert specific behavior. I’ve found a pretty simple way to do this type of testing in a reliable, consistent, and non-disruptive manner. Despite the fact that the technique leverages Java 1.5 built-in concurrency utilities, most of the engineers who have seen it are surprised and weren’t aware that such testing was so easy to implement.

Given the above service interface, here’s a test that will assert that no single article is given out to more than one invoker of the method findNextArticleForModeration(). The scenario we’re simulating is 10 users feverishly moderating a queue of 250 articles as quickly as possible.

ArticleServiceImplTest.java – Test to invoke service concurrently

...
public void findNextArticleForModerationStressTest() throws Exception
{
    final int ARTICLE_COUNT = 250;
    final int THREAD_COUNT = 10;
 
    // Create test data and callable tasks
    //
    Set<Article> testArticles = new HashSet<Article>();
 
    Collection<Callable<Article>> tasks = new ArrayList<Callable<Article>>();
    for (int i = 0; i < ARTICLE_COUNT; i++)
    {
        // Test data
        testArticles.add(new Article());
 
        // Tasks - each task makes exactly one service invocation.
        tasks.add(new Callable<Article>()
        {
            public Article call() throws Exception
            {
                return articleService.findNextArticleForModeration();
            }
        });
    }
    articleService.createArticles(testArticles);
 
    // Execute tasks
    //
    ExecutorService executorService = Executors.newFixedThreadPool(THREAD_COUNT);
    // invokeAll() blocks until all tasks have run...
    List<Future<Article>> futures = executorService.invokeAll(tasks);
    assertThat(futures.size(), is(ARTICLE_COUNT));
 
    // Assertions
    //
    Set<Long> articleIds = new HashSet<Long>(ARTICLE_COUNT);
    for (Future<Article> future : futures)
    {
        // get() will throw an exception if an exception was thrown by the service.
        Article article = future.get();
        // Did we get an article?
        assertThat(article, not(nullValue()));
        // Did the service lock the article before returning?
        assertThat(article.isLocked(), is(true));
        // Is the article id unique (see Set.add() javadoc)?
        assertThat(articleIds.add(article.getId()), is(true));
    }
    // Did we get the right number of article ids?
    assertThat(articleIds.size(), is(ARTICLE_COUNT));
}
...

The test starts off by creating 250 test articles to be moderated. It also creates 250 ‘tasks’, each designed to make a single service invocation of findNextArticleForModeration(). The real magic happens in Executors.newFixedThreadPool() and executorService.invokeAll(). The first creates a new ExecutorService backed by a thread pool of the specified size. This is a generic ExecutorService that is designed to churn through tasks using all of the threads in the pool. invokeAll blocks until every task has finished executing. In this test, 10 threads will rip through 250 tasks, each making a single call to our service and capturing the result of that call. Each task execution results in a Future, which is a handle to the results of the task (and more).

Iterating over each resulting future, we make several assertions. The most important one is the last, where we assert that every task is given a unique Article. Thanks to the natural semantics of Set, this is easy to do in an elegant way. Another useful, though unexpected, feature is that if an exception occurs during the task execution, an ExecutionException will be thrown when get() is called on the corresponding Future. If our service fails for some reason, the test will fail because no exceptions are expected.

This technique makes simulating a multi-threaded environment in a test easy and readable. It’s important to only use this technique when it’s really necessary. The resulting test is more of an integration test than a unit test, and its run time is orders of magnitude more than a unit test, so overuse of the technique will artificially inflate the time it takes to runs the tests. After I’ve finished working on the component under test, I will reduce the test-data size and thread count to a level that the test still provides value, but is no longer a stress test (e.g. 10 articles and 2 threads). The next time the component is being worked on, the developer can crank up the values and run the tests to be confident that the behavior isn’t broken.

The complete source for a working example of this technique is available here. You’ll need Maven (or IntelliJ IDEA 7.x) to build and run the test. By default, the tests run against an in-memory H2Database instance, but if you look at application.properties you’ll see configurations for PostgreSQL and MySQL as well.

Happy testing!

Problem

In v2.2 DBUnit introduces some new abstractions, one of which is the IDatabaseTester. Whether it’s by design or by accident, the default implementation closes connections after executing its operations (more info). The end result is that in our unit tests, the database connection is closed before the test can run.

Solution

I’ve created two simple util classes for loading test data into a database without closing the connection:

/** A helper for loading data sets into unit tests. */
public class DatabaseUtils {
    public static void loadDataSet(Class clazz, final DataSource dataSource) throws Exception {
        IDataSet dataSet = new FlatXmlDataSet(TestUtils.datasetInputStream(clazz));
        IDatabaseTester tester = new ExistingConnectionDatabaseTester(dataSource);
        tester.setDataSet(dataSet);
        tester.onSetup();
    }
}

/** A special DatabaseTester that doesn’t close the connection when its done. */
public class ExistingConnectionDatabaseTester extends AbstractDatabaseTester {
    private DataSource dataSource;
 
    public ExistingConnectionDatabaseTester(DataSource dataSource) {
        super();
        this.dataSource = dataSource;
    }
 
    public IDatabaseConnection getConnection() throws Exception {
        return new DatabaseConnection(DataSourceUtils.getConnection(dataSource));
    }
 
    public void closeConnection(IDatabaseConnection connection) throws Exception {
        // Don't close that connection!
    }
}

The first of these depends on another test-related utility, TestUtils.java (see below), which converts a class to a path. The second depends on Spring’s DataSourceUtils.

/**
 * A set of utilities that generate paths from classnames.  This is useful when
 * making reference to resources used by test cases, when the resources are
 * located in a directory path matching the class' package.
 */
public class TestUtils {
 
    public static final String TEST_PREFIX = "src/test/resources/"; // Maven2 default
 
    public static String pathString(Class clazz) {
        return pathString(TEST_PREFIX, clazz, "");
    }
 
    public static String pathString(Class clazz, String resource) {
        return pathString(TEST_PREFIX, clazz, resource);
    }
 
    public static String pathString(String prefix, Class clazz, String resource) {
        prefix = (prefix != null ? prefix : "");
 
        StringBuffer sb = new StringBuffer();
        sb.append(prefix);
 
        if (!prefix.endsWith("/")) {
            sb.append("/");
        }
 
        sb.append(ClassUtils.classPackageAsResourcePath(clazz));
        sb.append("/");
        sb.append(resource);
 
        return sb.toString();
    }
 
    public static InputStream pathInputStream(Class clazz, String resource)
        throws FileNotFoundException {
        return pathInputStream(TEST_PREFIX, clazz, resource);
    }
 
    public static InputStream pathInputStream(String prefix, Class clazz, String resource)
        throws FileNotFoundException {
        return new FileInputStream(pathString(prefix, clazz, resource));
    }
 
    public static InputStream datasetInputStream(Class clazz) throws FileNotFoundException {
        return pathInputStream(TEST_PREFIX, clazz, ClassUtils.getShortName(clazz) + ".xml");
    }
}

These three classes give us everything we need to load data into our test database using DBUnit. And TestUtils can be used to load other test resources from the classpath as well, like images, documents, CSVs, etc.

Here’s an example of how it can be used in a DAO test based on Spring’s test classes.

public class ArtistHibernateDaoTest extends AbstractAnnotationAwareTransactionalTests {
 
    private ArtistHibernateDao artistDao;
 
    public void setArtistDao(ArtistHibernateDao artistDao) {
        this.artistDao = artistDao;
    }
 
    protected String[] getConfigLocations() {
        return new String[]{"applicationContext-database.xml", "applicationContext-hibernate.xml"};
    }
 
    protected void onSetUpInTransaction() throws Exception {
        DatabaseUtils.loadDataSet(getClass(), getJdbcTemplate().getDataSource());
    }
 
    public void testFindAllNames() {
        List<String> names = artistDao.findAllNames();
        assertEquals(3, names.size());
        assertTrue(names.contains("The Cure"));
        assertTrue(names.contains("Depeche Mode"));
        assertTrue(names.contains("New Order"));
    }
}

onSetUpInTransaction() loads the data set using the convention of (packagename).(ClassName).xml before each test (e.g. com.c5.PizzaTest loads the file com/c5/PizzaText.xml on the classpath). In this case the data is loaded in the same transaction as the test so that no clean-up is necessary when the test is completed. These tests run fast!

We’ve been talking about moving away from our custom DatabaseTestCase class hierarchy. The above functionality in conjunction with Spring’s test hierarchy could be a good start.

On a somewhat related note, there is a new unit testing framework called Unitils which does this sort of thing and much more. I haven’t used it but it sounds interesting.

Christian

Imagine we have a test fixture, BlogEntryController, which depends on a single service, BlogService. Because we want to focus our testing on the fixture, we’ll inject a mock object as the service.

//follow EasyMock's convention of static imports
import static org.easymock.EasyMock.*
...
BlogEntryController controller = new BlogEntryController();
BlogService mockBlogService = createMock(BlogService.class);
controller.setBlogService(mockBlogService);

The next step is to record the expectations about the interactions between the fixture and its collaborator. In the record phase, we simply interact with the mock to record these expectations:

//record phase
BlogEntry blogEntry = new BlogEntry();
mockBlogService.saveBlogEntry(blogEntry);
 
//replay phase
replay(mockBlogService);
 
//method under test
controller.handleRequest(request, response);
 
//verify
verify(mockBlogService);

By default, argument expectations are verified using the equals() method. For the previous expectation, when the controller code is run, EasyMock will assert that the BlogEntry passed in by the controller equals() that passed in during the record phase. If, for instance, the controller creates its own BlogEntry and BlogEntry has the default implementation of equals(), this expectation would fail. Fortunately, EasyMock provides other argument matchers which can verify arguments based on different criteria:

mockBlogService.saveBlogEntry(isA(BlogEntry.class));

By using the isA(Class clazz) argument matcher, one of several predefined in EasyMock, the code will verify that the argument passed from the controller to the mock service is an instance of the BlogEntry class. By relaxing our expectations, we’ve come up with a test case which would pass. However, generally speaking, we’ll probably want to test some more details about the BlogEntry. For instance, it’s common to want to verify the argument’s properties.

EasyMock has the facility to create custom argument matchers. It’s a somewhat heavy weight solution requiring the implementation of 3 methods: 2 for the IArgumentMatcher interface and one static method used during the record phase. This facility seems better suited to creating reusable argument matchers (akin to the predefined ones which come with EasyMock) rather than singular matchers relevant only to a particular test case.

To bridge the gap, I have created a generic argument matcher, EasyMockUtils.argAssert(Assertion<E> assertion), which abstracts away the required EasyMock interaction code, allowing one to more simply express, inline if desired, the argument expectations:

static import common.test.easymock.EasyMockUtils;
...
mockBlogService.saveBlogEntry(argAssert(new Assertion<BlogEntry>() {
    void check(BlogEntry blogEntry) {
        assertEquals("submitted blog entry body", blogEntry.getBody());
        assertNull(blogEntry.getDateCreated());
    })

The argAssert static method takes a genericized instance of the Assertion interface in which one can declare any number of assertions against the argument passed in during the replay phase. As an added benefit, the Assertion implementation could also be leveraged to modify the passed in argument. For instance, imagine the controller depends on the service setting the dateCreated property on the passed in bean. We can ensure our mock replicates that behavior:

mockBlogService.saveBlogEntry(argAssert(new Assertion<BlogEntry>() {
    void check(BlogEntry blogEntry) {
        assertEquals("submitted blog entry body", blogEntry.getBody());
        assertNull(blogEntry.getDateCreated());
        //mimic the behavior of the true service
        blogEntry.setDateCreated(new Date());
    })

Mocking to this detail could trigger a warning regarding the level of coupling between the test, the fixture, and the collaborator; however, I have seen situations where it can make the difference between being able to use a mock object for testing or not. Mockist testers in general must be wary of unnecessarily deep coupling where, for instance, a refactor of the fixture/collaborator interactions could result in as much time spent updating test code even if the test’s high level assertions remain unchanged. However, used attentively during test driven development, mock objects can help a programmer focus on keeping those interactions simple and clearly defined. Furthermore, testing with mock objects helps one to focus on unit testing the fixture at hand. In that situation, the generic custom argument matcher provides a simplified tool for expressing assertions about that code.

And here’s the code. The generic interface which allows one to specify argument assertions:

Assertion.java

package common.test.easymock;
 
public interface Assertion<E>
{
    /**
    * Allows specifying varying assertions about an argument.
    * @param argument the argument being check
    * @throws Error throw an exception in the case of an assertion failure
    */
    void check(E argument) throws Error, Exception;
}

A simple container for the static method used to declare the argument matching assertion during the mock record phase.

EasyMockUtils.java

package common.test.easymock;
 
import org.easymock.EasyMock;
 
public class EasyMockUtils
{
    public static <E> E argAssert(Assertion<E> assertion)
    {
        EasyMock.reportMatcher(new ArgumentAssertion(assertion));
        return null;
    }
}

And the IArgumentMatcher implementation which ties the pieces together:

ArgumentAssertion.java

package common.test.easymock;
 
import org.easymock.IArgumentMatcher;
 
public class ArgumentAssertion implements IArgumentMatcher {
    private Assertion assertion;
    private Error assertionError;
 
    public ArgumentAssertion(Assertion assertion) {
        this.assertion = assertion;
    }
 
    public boolean matches(Object actual) {
        try
        {
            assertion.check(actual);
            return true;
        }
        catch (Error e)
        {
            assertionError = e;
            return false;
        }
        catch (Exception e)
        {
            assertionError = new Error(e);
            return false;
        }
    }
 
    public void appendTo(StringBuffer buffer) {
        buffer.append("argumentAssertion(exception ").append(assertionError).append(")");
    }
}

And of course a test:

ArgumentAssertionTest.java

package common.test.easymock;
 
import static common.test.easymock.EasyMockUtils.argAssert;
import junit.framework.TestCase;
import static org.easymock.EasyMock.*;
 
public class ArgumentAssertionTest extends TestCase
{
    public interface TestInterface
    {
        void testMethod(String arg);
    }
 
    TestInterface mock;
 
    protected void setUp() throws Exception
    {
        super.setUp();
        mock = createMock(TestInterface.class);
    }
 
    public void testArgumentAssertionSucess()
    {
        //record expected behavior
        mock.testMethod(argAssert(new Assertion<String>() {
            public void check(String argument)
            {
                //do nothing - should pass
            }
        }));
 
        //should get no errors
        replay(mock);
        mock.testMethod("test");
        verify(mock);
    }
 
    public void testArgumentAssertionFailure()
    {
        //record expected behavior
        mock.testMethod(argAssert(new Assertion<String>() {
            public void check(String argument)
            {
                fail("explicitly fail assertion");
            }
        }));
 
        replay(mock);
 
        boolean testSucceeded = false;
 
        try
        {
            mock.testMethod("test");
            verify(mock);
            testSucceeded = true;
        }
        catch (AssertionError e)
        {
            //should get this error
        }
 
        if (testSucceeded)
            fail("should have gotten a failure");
    }
}

The error looks like this:

WARNING - TABLE.COLUMN data type (16, 'BOOLEAN') not recognized
and will be ignored. See FAQ for more information.

The solution outlined here is straightforward. You need to create a new data type factory that extends the DBUnit class DefaultDataTypeFactory. This new class just handles the SQL type Types.BOOLEAN as a special case. The code follows:

HsqlDataTypeFactory.java

package company.project.test;
 
import org.apache.commons.logging.*;
import org.dbunit.dataset.datatype.*;
 
import java.sql.*;
 
public class HsqlDataTypeFactory
  extends DefaultDataTypeFactory
{
  private static final Log log = LogFactory.getLog(HsqlDataTypeFactory.class);
 
  public DataType createDataType(int sqlType, String sqlTypeName)
    throws DataTypeException
  {
    if (sqlType == Types.BOOLEAN)
    {
      return DataType.BOOLEAN;
    }
 
    return super.createDataType(sqlType, sqlTypeName);
  }
}

Then, in order to use this data type factory, just set a property on the IDatabaseConnection DBUnit object in your code (here is an example method):

  protected IDatabaseConnection getConnection() throws Exception
  {
    IDatabaseConnection connection =
      new DatabaseConnection(dataSource.getConnection());
    DatabaseConfig config = connection.getConfig();
    config.setProperty(DatabaseConfig.PROPERTY_DATATYPE_FACTORY,
                       new HsqlDataTypeFactory());
 
    return connection;
  }