Property-based testing in Go

This posts hopes to illustrate

  • What property based testing is and why it is an important tool to compliment your existing tests
  • How easy it is in Go

Your current tests

The usual examples illustrating TDD could be described as example based tests; where you write tests by providing example inputs and expected outputs.

func TestAddingExample(t *testing.T) {
    result := add(3, 2)
    if result != 5 {
        t.Error("3 plus 2 is 5 but i got", result)
    }
}

When you realise that they are just examples, you see your code perhaps isn’t as well-tested as you think.

Wouldn't it be nice if it were easy to provide thousands of different test inputs for your tests?

You may have already heard about property-based testing and think its about testing your code with randomly generated data to exercise different scenarios; but it actually requires a shift in mindset to get the most out of it.

Specifications, not just lots of data

Property-based testing is called that because in order to write the tests you have to think about the properties of your domain in order to write effective tests.

 An example with addition

We will test an add function, which just adds 2 integers together.

func add(x, y int) int {
    return x + y
}

The tricky thing when you feed in random data, is how do you know what your expected result should be?

func TestAddition(t *testing.T){
  x := getRandomInt()
  y := getRandomInt()

  expected := x + y

  if expected != add(x, y){
    // oh no..
  }
}

You may be tempted to do the above, but in effect you’re cheating because you’re using the implementation to verify itself.

Pretend that you’re testing a more complicated function. You know you shouldn’t do the following:

func TestComplicatedThing(t *testing.T){
  x := getRandomInt()
  y := getRandomInt()

  expected := complicated(x, y)

  if expected != complicated(x, y){
    // oh no..
  }
}

This isn’t really testing anything, if you change the implementation of complicated it will continue to pass its tests even if its behaviour is incorrect.

(This is a common problem with bad tests where the writer has re-implemented the unit they are testing, in the test!)

But what do you do? How can you verify your function on random inputs?

The properties of addition

What is there to addition anyway? Well, there’s a few things:

  • when you add zero to a number, you get the same number back again
  • the order of the inputs does not matter

Let’s write some property based tests based on these laws of addition.

func TestAddingZeroMakesNoDifference(t *testing.T) {
    /*
    Create an assertion, which is a function that takes N inputs of
    random data and returns true if the assertion passes. 

    In this case, we're saying take any random integer (x)
    If you add 0, it should equal x
    */
    assertion := func(x int) bool {
        return add(x, 0) == x
    }

    // Run the assertion through the quick checker
    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

func TestAssociativity(t *testing.T) {
    assertion := func(x, y, z int) bool {
        return add(add(x, y), z) == add(add(z, y), x)
    }

    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

The quick.Check function will run your assertion 100 times (by default), feeding in randomly generated data for the inputs required. If it fails at any point then you will be told for what inputs your function fails on so you can go reproduce it and fix.

Notice how:

  • By asserting on properties rather than known values we don’t have to "cheat" on the tests like I did earlier.
  • We have now described what addition is in terms of its properties, which is more expressive than the example based test from earlier.
  • We now have a more robust test suite, able to run thousands of different inputs in milliseconds.

Thinking abstractly about your domain

Once you take a step back and really think about the domain of your code, you may discover some unexpected properties that the examples you use in your general TDD flow wouldn’t find.

I imagine this is how a lot of QAs think. They tend to be a bit detached from the code we write and really think about the domain and how things can fall apart. Property based testing allows us to capture these kind of rules as tests.

Going beyond simple types

Property-based testing relies on generators for types to create random data for your tests. It’s simple to imagine how this works for integers but what about your own types?

type LatLong struct{
    Lat float64
    Long float64
}

func (l LatLong) Generate(rand *rand.Rand, size int) reflect.Value{
    randomLatLong := LatLong{
        Lat: rand.Float64(),
        Long: rand.Float64(),
    }
    return reflect.ValueOf(randomLatLong)
}

func TestLatLongIsAlwaysTrue(t *testing.T){
    assertion := func(x LatLong) bool {
        t.Log("Random LatLong:", x)
        //todo: Do some interesting assertions!
        return true
    }

    if err := quick.Check(assertion, nil); err != nil {
        t.Error(err)
    }
}

All you have to do is implement the Generator interface for your type. Go will provide you with a rand to help you generate random data.

Conclusion

To write effective property-based tests you have to take a step back and think deeply about the rules of your domain.

Once you gain this deep understanding you can then express them very easily with property-based tests which will help you find bugs before you get to production.

The tooling in Go facilitates easily writing these specifications, even for your own types so there’s no excuse not to try it in your current project.

Notes