Chris James - Software Engineer and other things

Property-based testing in real life

27 August 2016

Most property-based test examples you see on the web are contrived scenarios and people can be dismissive of them, claiming that they aren't actually practical. This post will have a simple example applied to one of my projects which will hopefully illustrate their value.

This is a follow up to Property-based testing in Go and if you're not familiar with these kind of tests I suggest you read that first, it's a quick read.

Although the examples are in Go there are QuickCheck frameworks for most mainstream programming languages and the concepts described here will still apply.

This post will:

mockingjay server

It's important to have a good knowledge of the domain when writing property-based-tests, the following is an overview of the domain we will be testing.

mockingjay-server (MJ) is an application which is a lightweight HTTP server driven from configuration.

 - name: My very important integration point
     uri: /hello
     method: POST
     body: Chris
     code: 200
     body: '{"message": "hello, Chris"}'
       content-type: application/json

MJ has two "modes":

Server mode

Serves responses matched to requests that you define.

Compatibility or CDC mode

Take that same config, executing each request against a given URL and checking the response is compatible with the response in configuration. This is known as a consumer-driven contract (CDC).


When writing integration tests against a HTTP service you will usually make a fake server (or some kind of stub around HTTP) to write tests against. You will then make your tests pass and be happy right?


In both of these situations your build would be green but your software is broken.

mockingjay allows you to make a server and easily verify that it is equivalent to what you're testing against. You can then distribute this configuration as a CDC for the maintainer of the downstream service, so they dont accidentally break the service for your use case. The wiki goes into this more.

There is an inherent coupling between HTTP integration tests and consumer driven contracts. MJ leverages this in a single config.

A property of mockingjay

To write a property-based test we need to identify a property to throw lots of auto-generated data at, to make sure the property holds true.

A property of MJ is:

MJ should always be compatible with itself

Here's my thinking:

If it doesn't it either means there is a flaw in the CDC algorithm or in the way the server is behaving.

I have lots of example based tests for this but if I invest time writing a property-based test I can be really confident MJ is working.

Create a generator

For all but the basic types you will need to create a Generate method for the input type in your test. This will allow the quickcheck package to create thousands of different data points to check the property against.

I made a simple one to start with and other HTTP things like headers, forms, etc can be added later.

func (r FakeEndpoint) Generate(rand *rand.Rand, size int) reflect.Value {
    randomMethod := httpMethods[rand.Intn(len(httpMethods))]

    req := Request{
        Method: randomMethod,
        URI:    "/" + randomURL(rand.Intn(maxURLLen)),

    res := response{
        Code: rand.Intn(599-100) + 100,

    return reflect.ValueOf(FakeEndpoint{
        Name:     "Generated",
        Request:  req,
        Response: res,

FakeEndpoint is a representation of the config from earlier.

Test the property

Here's what the test looks like

func TestItIsAlwaysCompatibleWithItself(t *testing.T) {

    compatabilityChecker := NewCompatabilityChecker(noopLogger, httpTimeout)

    assertion := func(endpoint FakeEndpoint) bool {

        // Start an MJ server with the random configuration
        mjSvr := NewServer([]FakeEndpoint{endpoint}, false, ioutil.Discard)
        svr := httptest.NewServer(http.HandlerFunc(mjSvr.ServeHTTP))
        defer svr.Close()

        // Run CDC against "itself". An MJ server should always be compatible with itself.
        errors := compatabilityChecker.check(&endpoint, svr.URL)

        if len(errors) > 0 {
            t.Log("It wasn't compatible with itself")
            for _, err := range errors {

        return len(errors) == 0

    config := quick.Config{

    if err := quick.Check(assertion, &config); err != nil {

When I ran the test I was pleasantly surprised in that the CDC check failed.

Couldn't reach real server: Post 303 response missing Location header

This means the CDC tried to POST to the configured URL and Go's HTTP client returned an error.

This is quite exciting, I haven't seen MJ configured with any 3xx response codes so this points to some naivety in my code.

HTTP 303 is "See other" which relies on a location header to get redirected to another resource.

Investigating Go's HTTP client

Before writing any code for this I wanted to have a look at the Go source of http.Client to see exactly how it works.

By following a few of the function calls we can see if it's a POST or a PUT and the status is a HTTP Found 302 or See Other 303 then it will expect a location header that it can parse with req.URL.Parse.

I will have to add some additional validation to the configuration so that these rules can be respected or make Go's HTTP client not follow redirects.


The costs of writing property-based tests are low (this took me about 10 minutes) and can help give you a lot of confidence in your code (or not!).

What's great is that even on a very simple generator implementation the tests uncovered some bugs.

This style of testing can:

These things prove that property-based tests can improve the quality of your software.


Other examples of this style of testing in the real world:

QuickChecking Riak

In this talk John Hughes shows us how QuickCheck helped us to model Riak’s behaviour, improving understanding and revealing the occasional bug.

John Hughes - Testing the hard stuff and staying sane

Taking 3k lines of specification to create 20 lines of QuickCheck to test 1 million lines of code from 6 different vendors. Some real war stories in this video.