C5 Test Support new addition: FunctionalTestRunner

Christian Nelson ·

We’re always looking for new ways to test our applications and we’ve been trying a few new things on our projects. One of the recent additions is a JUnit test runner designed to help make writing and running functional tests easier. In Javaland, we use Selenium and/or HtmlUnit for our functional tests. These are the tests that run against a deployed application over the wire using a real or simulated browser. Most of our functional tests work the application in the same way a real user would, testing sequences of realistic activity and often touching a number of pages. Since our functional tests use either a real browser or a simulated one, Javascript is executed and assertions made on the results. This gives us greater confidence that our app is really working, end to end.

Here’s the high-level flow that the functional test runner provides:

  1. Load fixture data from a DBUnit dataset.
  2. Download and install the application server (if necessary).
  3. Start the application server (using Cargo).
  4. Deploy the application, waiting until it’s completely started.
  5. Run one or more functional tests (using your preferred testing framework – Selenium, HTMLUnit, etc)…
    If a test dirties the database in a manner that must be reset, the test class can be marked with the @DirtiesDatabase annotation. This will reload the database fixture and optionally restart the application.
  6. Shutdown the application server.

It’s easy to use… First add the maven dependency:

          <name>Carbon Five Public Repository</name>

And then write your first functional test:

public class FunctionalTestRunnerTest
    // This field is automatically injected by the test runner and references
    // the functional test properties (see below).
    private Properties properties;

    public void propertiesShouldBeSetByRunner()
        assertThat(properties, not(nullValue()));
        assertThat(properties.size(), greaterThan(5));

    public void applicationShouldBeDeployed() throws Exception
        // Test using your preferred framework... just using URL as an example.
        URL root = new URL(format("http://localhost:%s/", properties.getProperty("appserver.port")));
        String content = IOUtils.toString(root.openStream());
        assertThat(content, containsString("Orange"));

Lastly, create the configuration file that goes along with it. By default, it’s called functional-test.properties and lives at the root of the classpath (src/test/resources/ for maven users). Here’s an example:

# Application server
appserver.container = tomcat6x
appserver.installer = http://www.apache.org/dist/tomcat/tomcat-6/v6.0.20/bin/apache-tomcat-6.0.20.zip
appserver.port = 8080
appserver.logging = false

# Application configuration
app.war = ./gearlist-webapp/target/gearlist-webapp-1.0-SNAPSHOT.war
app.context = ROOT

# Database fixtures
fixture.restart_application = false
fixture.file = classpath:/database-fixture.xml

# Application configuration (not used by the functional test runner)
db.driver = com.mysql.jdbc.Driver
db.host = localhost
db.name = gearlist_func_test
db.url = jdbc:mysql://${db.host}/${db.name}
db.username = root
db.password =

Property values can reference other properties (e.g. db.url) and they’re all set as system properties so it’s easy to override your application properties for functional testing.

Both the database fixture loading and the application server steps can be customized by plugging in a new DatabaseFixtureLoader or ApplicationServerManager, respectively.

Since it’s a JUnit extension, functional tests run exactly the same whether they’re triggered from the IDE or the command line. This approach is a response to how clunky it felt using Maven to launch the application server before running the tests. Also, with only one master orchestrator, it’s easier to handle redeploying the application if necessary.

Want to see a real example? I’ve written a sample application called “gearlist” that’s available via public svn (press cancel if asked for authentication credentials). It includes several HtmlUnit and Selenium functional tests, as well as other types of tests using some of the other test utils I’ve written. Look for the readme for details.

Carbon Five Test Support is hosted on google code, where you can read what documentation there is and peruse the source.

There are a few things to work on, including better exception handling during initialization and some flexibility around the Cargo configuration, however, it’s pretty stable and there’s no reason not to use it now. Give it it try and give me your feedback. The plan is to tighten it up and write some documentation for the 0.9.2 release.

Christian Nelson
Christian Nelson

Christian is a software developer, technical lead and agile coach. He's passionate about helping teams find creative ways to make work fun and productive. He's a partner at Carbon Five and serves as the Director of Engineering in the San Francisco office. When not slinging code or playing agile games, you can find him trekking in the Sierras and playing with his daughters.