Why We Put Our Entire Payment Gateway in the Database: 5 Years of Statecharts at Kronor.io

The Payment Journey is a Minefield

Building a payment gateway is fundamentally a struggle against entropy. The journey of a payment is not as a simple API call, but as a treacherous path through a fragmented ecosystem of intermediaries. From the moment a customer clicks “Pay,” your request is tossed across a mixed-chain of legacy systems and modern APIs, each with its own idiosyncratic failure modes and timeouts:

The hard truth of payment engineering is that each one of these steps could fail. A network glitch at the PSP or a slow response from an issuing bank must be handled as an expected failure. The risk of improper handling of the errors is the appearance of “zombie” payments: charges that occurred in the real world but are missing from your records, or duplicate charges that inevitably end up as customer support requests and an experience of frustration.

If we break the process above into the database transactions that are required for the payment gateway to store the necessary state, it would take 3 different transactions:

What to the customer should look as a single atomic transaction, we are now forced to break into multiple transactions, all of which could fail for different reasons and leave the payment in an inconsistent state. What is key factor that is forcing us to break it up into multiple transactions instead of having just one?

If we identify key state changes, we realize that we are forced to label a new state each time we perform a foreign state change, that is, each time we make a call to an external system to alter the world.

This leave us with two important insights

1. We should minimize foreign state changes. That is, as much as possible, we should avoid making external requests to change data. Otherwise, we increase the number of places we need to save state and do possible error recovery.
2. Application flow is directed by data changes, not processes

State Machines

The most obvious, robust choice to handle these state changes is by creating a state machine. This is how the process would look like by making one:

The translations of our previous diagram to a state machines leaves us with new vocabulary. Now we have states, transitions and events that must happen during those transitions.

State Explosion

Anyone who has used state machines for some time knows that they are prone to a problem known as “state explosion”. Trying to model entire processes as state machines leads to unreadable representations.

For example, here’s the state machine used by the project pg_autofailover to track the state of a Postgres server inside a replication cluster:

(hint: don’t try to read this graph)

The interesting thing about the graph above is that there are not that many different states. What makes this image look like spaghetti is the number of possible transitions between the states, all of them possibly triggering a different event.

What if there was a way to tame this complexity?

Introducing Statecharts

Statecharts are a type of state machine that allows arbitrary nesting of smaller state machines. It’s a way of composing larger ones from many smaller ones. This is the solution to the state explosion problem.

The state machine we used to exemplify the state explosion problem could be simplified in a statechart like this:

Statecharts also introdue new vocabulary. We now have composite states, internal transitions, actions, and guards. This richer vocabulary allows us to encode more properties of the system inside the state machines and helps us model entire business process without losing track of the details.

From day one at Kronor, we knew we wanted to use statecharts to model our processes. What came out of this has been, in hindsight, core to the stability and efficiency of our solutions.

Let’s take a look at how we author our statecharts.

Authoring a Statechart

The interesting thing about statecharts is that they are a standard, in the same way that HTML is. There is an ecosystem around them, so you can author them, preview them and even run them in memory to test them out.

The W3C consortium standardized the SCXML format for this purpose, which is the one we use to author our statecharts at kronor. Here’s a sample of one of them:

We create or change Statecharts by changing the SCXML and visually validating the results as we edit them. The tooling around the standard makes it a breeze to share knowledge with team members about what happens in each step and also prevents many bugs by helping developers think-through all the edge cases.

In particular, one of the things that we have found extremely useful about statecharts is the ability to name “what happens when you enter or exit this state”. Whenever we need to go back to a process we haven’t touched in months, we start by reading the Statechart to remember what happens at each step.

The innovation: Running statecharts in the database

Most developers I know, when using state machines, write the code for expressing their state machine and driving it in application code. Some others opt for solutions like AWS Step Functions, which store both the state of the machines as well as the logic that transitions from one step to the next.

Remember when I said that we should minimize foreign state changes? Partitioning the state machine logic and the state where it lives would, by necessity, increase the number of foreign state mutations.

For example, imagine that your application receives a notification that a transaction has timeout out. Your application would need to first check the current state of the state machine associated with the transaction, run some logic to figure out what the next transition is, and store the the new state in the database, trigger any actions as a result of it, and finally save the state of those actions back to the database.

Each one of those interactions with the database could fail, and therefore this processes increase the surface area for error handling. We wanted to minimize that from the beginning, so we adopted a novel solution: run the state machine logic directly in the database where the rest of the application data lives.

So, after authoring a statechart using the SCXML language, we compile the representation down to SQL:

And the we expose an API as database functions to interact with the state machines:

Thanks to this API, a typical request to our application looks very simple: parse the input form the client, validate it and send a notification to the relevant state machine. The rest of the logic is handled by the statechart code, making our application code super simple. They all basically boil down to a single, basic SQL statament.

How do you interact with external services, then?

Naturally, you cannot make HTTP or other network requests from inside the database. At least that’s not something I would encourage. So, how does an application that has all of its logic running inside the database make any changes to the outside world?

Let’s first appreciate how normally other developers do it:

1. Get request from the client
2. Check the state in the database
3. Run the logic to calculate the next state
4. Make a http request to an external service if needed
5. Save the new state

As we saw before, this has the problem that any of the steps can fail. So a good developer would likely save the job of making the http request in a queuing system:

1. Get request from the client
2. Check the state in the database
3. Run the logic to calculate the next state
4. Send job to the queue system
5. Save the new state

But that does not fix anything, does it? The queuing step can still fail, and given how entropy works, you will have inconsistent state in the database because you do not know if the job was ever executed.

More experienced developers would implement a transactional-outbox pattern:

The idea is to store the job inside the database in the same transaction that the state change logic is runinng. That way, there can be no mismatch between what’s left to be done and the application state. A relaying system will then take jobs from the database and move them to the actually queuing system.

With statecharts running in the database, this pattern is enforced out of necessity. Developers don’t even have to think about it. It’s the only way of interacting with the external world. This made our application resilient against failures from day one.

Instead of using a relay process and a different queuing technology, we just used Postgres as a queuing system:

Our experience after 5 years using this

Reflecting on the last five years using statecharts in the database I would say that we have experiences these benefits:

1. They help create conversation with stakeholders. Creating a statechart requires thought and helps discover edge cases sooner.
2. Statecharts allow for quick prototyping. Initial ideas can be dumped as new requirements or new information is made available.
3. Encoding business process as statecharts help keeping the domain knowledge in an understandable format.
4. Statecharts do help with taming the complexity of huge state machines.
5. Running statecharts in the database completely eliminated any race-condition scenarios in our payment logic.
6. Debugging tricky scenarios becomes possible because all events sent to a state machine are recorded in the database. So it is possible to understand each of the steps the application took.
7. Application code keeps being very simple and succinct.
8. This architecture forces us to always implement asynchronous workflows. This has had a gigantic positive effect in our reliability.

All-in-all we consider this architecture one of the key reasons for our speed of delivery and the relatively very few defect rate we have seen in out code.

It’s Opensource!

We have made statecharts runner available as a set of Postgres functions available for everyone to use.

Check out our Github repository here: https://github.com/kronor-io/statecharts