A Look at Test Generation in Cucumber and ScalaCheck

Jared Carroll ·

In a typical agile project the test suite grows roughly twice as fast as the non-test code. As developers, our goal should be the minimum amount of test code that specifies the behavior of the app. This often leads to using “clever” metaprogramming techniques in order to reduce boilerplate and overall lines of test code. Metaprogramming works fairly well, however the tradeoff is usually a cryptic implementation and hard to debug test failures. Some testing frameworks however offer alternatives to metaprogramming. Two such tools are the Ruby gem Cucumber and Scala’s ScalaCheck.

Cucumber’s Scenario Outlines

Cucumber’s scenario outlines are templates for scenarios. They’re similar to using a templating library like Ruby’s ERB. To make a scenario outline, you first write a generic scenario template and then you specify some data for it. When executing the scenario outline Cucumber will generate and execute specific instances of the template.

Here’s an example from git_pivotal_tracker; a Ruby gem that integrates Pivotal Tracker with Git. This feature contains three scenarios for a few of the command line scripts the come with the gem. They’re all structurally similar and only differ in their specific values.

Feature: Getting Started
  In order to begin work on an unstarted Pivotal Tracker story
  As a developer
  I want to be able to start the latest unstarted feature,
  chore or bug in a new Git branch all from the command line

 Background: A git repo for a project in Pivotal Tracker
   Given the following git repo:
     | Name                | Pivotal Tracker Api Token/Project Id |
     | git_pivotal_tracker | 123456790/12345                      |

  Scenario: Begin Working on the Latest Feature
    When I run `git-feature`
      And I run `git branch`
    Then the output should contain "feature-12345-sample_feature"

  Scenario: Begin Working on the Latest Chore
    When I run `git-chore`
      And I run `git branch`
    Then the output should contain "chore-12346-sample_chore"

  Scenario: Begin Working on the Latest Bug
    When I run `git-bug`
      And I run `git branch`
    Then the output should contain "bug-12347-sample_bug"

Notice how all three scenarios are identical except for the command that’s run and the name of the expected Git branch. We can replace all three scenarios with a single scenario outline that uses a separate table containing the commands and expected Git branch names.

  Scenario Outline: Begin Working on a Latest Unstarted Story
    When I run `<command>`
      And I run `git branch`
    Then the output should contain "<git-branch-name>"

    Examples: Story Type Specific Commands
      | command     | git-branch-name              |
      | git-feature | feature-12345-sample_feature |
      | git-chore   | chore-12346-sample_chore     |
      | git-bug     | bug-12347-sample_bug         |

This is a much more concise specification. Cucumber is now generating multiple scenarios for us, reducing duplication and total lines of test code.

ScalaCheck’s Property-based Testing

Property-based testing is a technique in which you specify properties that the code under test must obey. ScalaCheck is a Scala library that allows you to specify and verify such properties. However in ScalaCheck, you write the property and ScalaCheck verifies it by generating the test cases for you.

Here’s a simple example specifying some basic properties of Scala’s List class.

import org.scalacheck._

object ListSpec extends Properties("List") {

  property("concatenation") =
    Prop.forAll {
      (xs: List[Int], ys: List[Int]) => (xs ::: ys).size == (xs.size + ys.size)
    }

  property("head") =
    Prop.forAll {
      (xs: List[Int]) => xs.head == xs(0)
    }

}

The two properties are organized in a ListSpec object allowing them to be run together. Nothing special about that; standard stuff in any testing tool. Each property contains an anonymous function that returns a Boolean i.e. a predicate function. This function must return true for every test case in order for the property to be considered satisfied. Let’s now take a closer look at each property.

  property("concatenation") =
    Prop.forAll {
      (xs: List[Int], ys: List[Int]) => (xs ::: ys).size == (xs.size + ys.size)
    }

This is a property for List concatenation. It’s declaring a predicate function that takes two lists of integers (the fact that they’re integers doesn’t matter for this property; we just need an actual type). The function verifies that “concatenating two lists should result in a new list with a size equal to the sum of the sizes of the two lists”.

Running this results in the following output from ScalaCheck:

+ List.concatenation: OK, passed 100 tests.

ScalaCheck just generated 100 test cases of randomly generated lists for the property (if you’re curious what those randomly generated lists look like see the docs for info on collecting generated test data). Our test suite just got 100 test cases for four succinct lines of code.

The second property was for accessing the head i.e. the first item in a List.

  property("head") =
    Prop.forAll {
      (xs: List[Int]) => xs.head == xs(0)
    }

This property states that “the head of a list should be the first item in the list”.

However running this results in a failure:

! List.head: Exception raised on property evaluation.
> ARG_0: List()
> Exception: java.util.NoSuchElementException: head of empty list

Our property failed when given an empty list because an empty list has no head. We need a more rigorous specification.

  property("head") =
    Prop.forAll {
      (xs: List[Int]) => (xs.size > 0) ==> (xs.head == xs(0))
    }

Our new specification uses ScalaCheck’s implication operator ==> to limit our original property to “lists with at least 1 item”. ScalaCheck will now only generate random lists with at least 1 item for this property.

Another run and we’re passing:

+ List.head: OK, passed 100 tests.

This is a really high-level overview of ScalaCheck. ScalaCheck’s feature set is extensive and fortunately well documented. For the record, ScalaCheck is a port of Haskell’s QuickCheck; so it should look pretty familiar to Haskell developers.

Property-based testing is a powerful tool in testing invariant properties of a class or function. One use case that comes to mind is using it to test domain model validations. For example, validating that an attribute value falls within a given range such as that a User object’s age is always within 18-40 years. Property-based testing could also be used by Design by Contract practitioners for ensuring the invariant portions of a class’s “contracts”.

Conclusion

By using the well known technique of templating Cucumber helps you write less but more concise tests. ScalaCheck’s property-based testing reduces your total lines of test code by generating tests based on high-level properties. Both of these techniques obviously aren’t applicable to every situation. It’s best to use them in combination with traditional testing techniques to reduce the size of and improve your test suite.