Production Mirror Testing with ReRe

0. Introduction

ReRe is a record-and-replay tool for Java. Here we introduce how to use it to create a test pipeline mirroring production data.

Before rolling out a new version of code, we want to test it as as throughly as possible. Having the ability to use actual production data to verify the results adds confidence to the test.

One method to achieve this is shadow running. By having the new version run alongside the prod version, we can monitor if there are any errors. Shadow running has a few problems. First, we cannot guarantee a 1–1 matching to prod, the slight time difference between the shadow run and the prod run may result in data changes. It’s possible to have different results while both are correct. Next, there is no automated testing. Often we can only hand pick a few runs, and check if the results are correct.

ReRe is able to eliminate the time difference problem by caching the data dependencies, we guarantee that operating data is the same. We also enable automated verification. This lets us construct a mirror testing pipeline of prod. We can stream the dependency replay, and verification data, and have unlimited test cases generated from prod.

1. The Replay data

The mirror data consists of two parts, the replay data and the verification data. The replay data is a cache of the dependencies the program interacts with. We assume that if all the clients return the same data, the main program will also follow the same execution path.

ReRe is pretty powerful in recording. It can record return values and also modifications to input parameters. For example both of the following signatures can be recorded with ReRe.

class DatabaseClient {
  byte[] readData(String url);
  void readData(String url, byte[] destination, int length);
}

Please see our other articles on how ReRe records execution data.

With the above recording, we can rerun the program. Additionally, we can make some slight changes to the program, and still rerun it on the same input data.

2. The verification data

Now the question comes to how do we verify the rerun is correct. We do this by providing a verify API.

<T extends Serializable, Comparable> verify(T object);

During record mode, this API puts the object into a list, then returns. During replay/test mode, this API compares the object with the serialized value in the list, the test fails if the values are different.

Thinking of a traditional unit test, as follows

DatabaseClient client = mock(DatabaseClient.class);

doReturn(...).when(client).methods(...);

Main main = new Main(client);

assertEquals(main.doSomething(), 0);
assertEquals(main.doSomethingElse(), 1);

The assertEquals() is replaced by a 1 variable verify, but it is comparing 2 variables across runs.

The mocking and doReturns, are automated by ReRe interception. At record time, ReRe monitors the return values and saves it to file. At replay time, the behavior is constructed from file.

Note that the verify() method can be put anywhere in your code, as long as you don’t move the line too far away between versions, and both record and replay calls verify with same order.

Planned feature: verifyStateChanges(Lambda(x))

Currently verify only compares 2 serializable objects. But theoretically, it is possible to verify state changes are the same. Formally, if you have a lambda that takes 1 parameter, you can verify the state changes on X is the same, when the lambda is called.

3. Test Failures

Other than the application errors, ReRe will throw 2 additional errors at replay phase.

VerificationFailedException

Thrown with the verify() receives a different value than the recorded value.

OperationNotRecordedException

The replay is only resilient to “small” code changes. This small means the call pattern to the dependencies must remain the same. If at record time, only database.getPerson() was invoked, but at replay time, database.getPerson() and database.getCompany() are both invoked, there would be no record data to return. This would cause the OperationNotRecordedException.

This does not necessarily mean that the new code is wrong, it just means the test pipeline is unable to test this code change.

4. Conclusion

The record and replay algorithm by ReRe can create unlimited production tests, without affecting the production outcome. Consider using it to add deployment confidence.

Notes:

ReRe is still a new project, there maybe many bugs. Don’t hesitate to reach out for fixes!

Checkout the GitHub repo, or just download from Maven!