Property-Based Testing in Elixir

Brief: This is a story about a rookie alchemist to learn property testing in Elixir. Welcome to share my experience.

Maybe we should consider about property based testing.

In a retrospective meeting, a colleague said this. So the story began…I knew nothing about it at that time.

What’s property-based testing?

First of all I need more information about it. Is it cool stuff? Or just another buzzword?

This is my first impression after gathering information: it’s a kind of random generative testing. It confused me, what’s the difference with what we already have?

While I was trying to get a deeper understanding, I learned about a new term: shrink. It means that property-based testing can tell us what’s the minimal case to reproduce the error. Magic! Is there a rubber duck inside it?

Maybe I know why I should use property-based testing…

Sounds great. Before that, we need to write so many test cases, and consider about corner cases. Now what we need is to write a property test, and r̶u̶b̶b̶e̶r̶ ̶d̶u̶c̶k̶ ̶w̶i̶l̶l̶ ̶d̶o̶ ̶a̶l̶l̶ ̶t̶h̶i̶n̶g̶s̶ ̶f̶o̶r̶ ̶u̶s̶ the test framework can generate many cases including corner cases for us. If there is anything wrong, it will tell us what’s the minimal case to reproduce the error.

Knowledge as action

To demonstrate property testing, I implemented a simple queue in Elixir. It should give us the same result with Erlang’s build-in queue.

[embed]

By default it will repeat the checking 100 times. In our example it passed in 3.1 seconds.

> Heaven is near…?

Everything is perfect. What I need in the future is 5~10 lines of code of testing, and r̶u̶b̶b̶e̶r̶ ̶d̶u̶c̶k̶ ̶w̶i̶l̶l̶ ̶s̶a̶v̶e̶ ̶t̶h̶e̶ ̶w̶o̶r̶l̶d̶ property testing will handle everything else for us.

But you know, as an engineer, it’s too smooth to believe. In order not to be a fool, I added a test for 87 and it should fail…because I am a smart guy.

[embed]

Oh No! Am I a fool? I repeated it over 10 times and it always passed. How cruel this world is...

After I changed the integer range to 0~100, I finally got the test failure in 80% (8 in 10 times).

> So…is property testing useful?

Now we know that property testing is a kind of random generative testing with shrinking ability. It generates many random test cases including common edge cases such as nil value, empty list and so on. If it finds anything wrong during the testing, it will try to shrink the errors and find the minimal set which causes the problem. The property testing can test more and more different cases over time, maybe it could find something unexpected. But if we already know the edge case, and the case is not a common one (e.g., the case of 87), we should add a unit test for that case along with property testing. Then the world will be beautiful.

How to do property testing?

1. We need a property testing framework. In Elixir we have StreamData.

2. Suitable scenario. Key idea is about property or behavior. If there are some functions with specific properties, it’s time to use property-based testing. For example, our calculator should handle all decimal values, or the cast method should always convert a decimal to string.

Simple explanation for Elixir’s StreamData

In Elixir it’s quite simple and easy to understand how to write a basic property test like the example below.

[embed]

First line property means we want to use property testing (it will be test in unit testing). Focus on the second line. The term() is a generator provided by the property testing framework: StreamData. It provides many different kinds of generators, i.e.,term()that can generate all types of data. We can switch different generator according to the behavior of our target. If the target method only accepts integer, we can use integer() instead.

We can regard check all as a for loop. Literally what it does is iterating over all generated items. Then we can test our logic by each item.

Patterns in property testing

There are many articles about patterns. Here I only list the most commonly used in my opinion.

1. Round-Tripping

a.k.a. Encoder/Decoder. For example, we have float_to_string() and float_from_string(). We can generate random floating numbers, convert them to string then convert them back, the value should remain equal.

assert term |> encoder() |> decoder() == term

2. Oracle Model

Here is not that Oracle. Imagine that we have a correct but poorly performant model. We want to implement a better one that has to always give us the same result with the original one.

assert oracle_model(term) == new_model(term)

3. Smoke Testing

a.k.a Build Verification Testing. The purpose is to ensure the functionality of our method is working. For example, our method should always return :ok or:error, then we can have property testing like this:

result = my_code(term)

assert result in [:ok, :error]

If it passes, it means our method works (no smoke or on-fire).

4. Stateful Testing

If we have some stateful functions in our system, such as Memcached, Redis, ets or GenServer in Elixir, we can use property testing as below:

[embed]

References

These links help me a lot. I hope they also help you.

1. Introduction to Property Based Testing
2. Property-based or generative testing — insights from European Testing Conference
3. Property-based Testing is a Mindset — Andrea Leopardi — ElixirConf EU 2018
4. Property-based Testing Patterns

We want you

Flipay is looking for iOS and Android developers. If you are interested in us, feel free to contact us: https://jobs.blognone.com/company/flipay-co-ltd