Building a Payment Disbursement Platform with a Finite State Machine at Scale

Finite State Machine

“In financial systems, determinism is often more valuable than speed”

During my time at an InsurTech company, I worked on a centralized payment disbursement platform responsible for moving money across multiple business domains.

Insurance claim settlements were one of the primary consumers of the platform. Once a customer’s claim was approved by the claims system, a disbursement request was registered and handed over to the payment platform for execution.

However, claims were only one part of the story.

The same platform was responsible for:

• Insurance claim settlements
• Agent commission payouts
• Partner settlements

Whenever an insurance agent sold a policy, the commission payout was routed through the same payment infrastructure.

As the number of consumers grew, the challenge shifted from simply processing payments to ensuring correctness under failure.

The real engineering challenge was handling everything that could happen after a payout request was created.

Consider a few scenarios:

• The payment gateway API times out.
• The payment provider is unavailable due to scheduled maintenance.
• The payment provider does not have sufficient balance to initiate the payout.
• Network failures occur midway through processing.
• Provider callbacks arrive multiple times.
• Provider callbacks arrive out of order.
• Worker processes crash during execution.

These are not edge cases.

In a sufficiently large system, these are normal operating conditions.

The challenge was ensuring that every payout remained in a valid, auditable, and recoverable state regardless of infrastructure failures, third-party outages, or operational incidents.

To solve this, we built the heart of the payment platform around a Finite State Machine (FSM).

Rather than allowing business services to directly orchestrate payouts, every disbursement flowed through the FSM service, which became the single source of truth for payout lifecycle management.

Why a Dedicated FSM Service?

The FSM was implemented as its own microservice.

Every disbursement request carried metadata identifying:

• Originating service
• Service reference number
• Disbursement identifier
• Amount to be disbursed

Authentication between services was performed using Basic Authentication, allowing us to identify exactly which service initiated a payout request.

The FSM service became responsible:

• Managing payout lifecycle
• Validating state transitions
• Persisting transition history
• Publishing state change events
• Enforcing approval policies
• Recovering stuck transactions
• Preventing duplicate execution

This created a clean separation between systems generating payouts and the platform responsible for executing them.

FSM as a Policy Enforcement Layer

Initially, the FSM existed to orchestrate payout execution

Over time, however, it evolved into something more important

As the business grew, new controls were introduced around money movement

Certain payouts required manual approval based on configurable business rules such as:

• Agent name
• Beneficiary bank
• Transfer amount
• Daily payout limits
• Monthly payout limits

Interestingly, many of these requirements did not originate from technical constraints.

They were business-driven controls introduced to reduce operational risk, improve governance, or satisfy internal approval processes.

This meant the backend platform needed to remain flexible enough to accommodate new approval policies without requiring major architectural changes.

Instead of scattering these rules across multiple services, we centralized them within the FSM.

A payout could remain in the PENDING state until all required approval conditions were satisfied.

Only then would the FSM allow a transition into the APPROVED state.

The FSM therefore evolved beyond a workflow engine and became a centralized policy enforcement layer governing money movement across the organization.

Modeling the Payout Lifecycle

The payout lifecycle was modeled as a state machine.

Payout State Transition Diagram

The most important design principle was simple:

A payout should never exist in an ambiguous state.

At any point in time, the system should be able to answer:

“What is the current status of this payout?”

Green Channel: The Happy Path

Most payouts followed the happy path

The flow looked like this:

Init -> The payout request is created

VALIDATED -> Beneficiary information and payout details are verified.

PENDING -> Approval policies and business controls are evaluated

APPROVED -> The payout is authorized for execution

PROCESSED -> The payment provider acknowledges the request

COMPLETED -> The provider confirms settlement

REJECTED -> INVALID BANK DETAILS

FAILED -> MANUALLY FAILED/FRAUD TRANSACTIONS

Red Channel: Designing for Failure

The real value of the FSM emerged during failure scenarios.

Validation Failure

INIT
 ↓
INVALIDATED

Bank account verification fails.

The payout never reaches the payment provider.

Approval Rejection

PENDING
 ↓
REJECTED

Business policies or manual reviews prevent execution

Settlement Failure

PROCESSED
 ↓
FAILED

The request was accepted by the provider but ultimately failed during settlement.

Rather than treating failures as exceptions, we modeled them as explicit states.

This made operational behavior predictable and easier to reason about.

Payment State Machine

Explicit Transition Definitions

One of the biggest advantages of FSM-based systems is that business rules become explicit.

A simplified version of our transition definitions looked like this:

TRANSITIONS = [
    {'trigger': 'validate', 'source': 'init', 'dest': 'validated'},
    {'trigger': 'not_valid', 'source': 'init', 'dest': 'invalidated'},
    {'trigger': 'mark_pending', 'source': 'validated', 'dest': 'pending'},
    {'trigger': 'reject', 'source': 'pending', 'dest': 'rejected'},
    {'trigger': 'approve', 'source': 'pending', 'dest': 'approved'},
    {'trigger': 'process', 'source': 'approved', 'dest': 'processed'},
    {'trigger': 'complete', 'source': 'processed', 'dest': 'completed'},
    {'trigger': 'fail', 'source': 'approved', 'dest': 'failed'},
    {'trigger': 'fail', 'source': 'processed', 'dest': 'failed'}
]

This ensured invalid transitions could never occur.

For example:

• COMPLETED → APPROVED
• FAILED → PROCESSING
• REJECTED → VALIDATED

were impossible by design.

The FSM became the guardian of business correctness.

Event Driven Execution

The FSM was intentionally designed as an orchestrator rather than an executor.

Instead of performing business actions directly, it queued asynchronous work.

def queue_for_validation(self):
    validate_payout.delay(self.payout.id)

This approach provided:

• Loose coupling
• Independent scaling
• Retry capabilities
• Better resiliency

Each stage of the workflow could evolve independently without affecting the orchestration layer.

Preventing Duplicate Processing (Outbox Pattern)

One of the most important requirements in any payment system is ensuring that a payout is never processed twice.

During callback storms, retries, or concurrent worker execution, it is surprisingly easy to trigger duplicate processing.

To protect against this, we maintained a short-lived centralized in-memory cache like Redis that tracked payouts currently being processed

if payout_inprogress_queue.get(payout.id):
    pass
else:
    payout_inprogress_queue.set(
        payout.id,
        "approved",
        CacheSeconds
    )
    payout_transition.approve()

Before processing a payout, the FSM checked whether another worker was already handling it.

If the payout existed in the cache, processing was skipped.

Combined with persisted state transitions, this mechanism significantly reduced the risk of duplicate execution.

NOTE: This will not 100% gurantee the duplicate payment

Persistent State History

Every state transition was recorded.

def change_payout_state(self):
    self.payout.change_state(self.state)
    PayoutState.record_transition(
            self.payout.id,
            status=self.state
        )

This gave us:

• Full auditability
• Historical visibility
• Easier debugging
• Operational transparency

When investigating incidents, we could reconstruct the exact lifecycle of a payout from creation to settlement.

For financial systems, this level of visibility is invaluable.

Reconciliation: Recovering Stuck Transactions

• Workers crash
• Provider outages occur (Provider Balance is low)

To recover from these scenarios, we implemented reconciliation jobs

The reconciler periodically scanned payouts that were still in intermediate states and attempted to move them forward

@classmethod
def reconcile(cls, start_date=None, end_date=None):
    payouts = Payout.query
    if(start_date):
            payouts = payouts.filter(
                Payout.created_at >= start_date
            )
        if(end_date):
            payouts = payouts.filter(
                Payout.created_at <= end_date
            )
        payouts = payouts.filter(
            Payout.status.in_(
                ('init', 'validated',
                 'pending', 'approved',
                 'processed')
            )
        ).all()
        for payout in payouts:
            try:
                payout_transition = cls(payout)
                payout_transition.process_for_next_state()
            except Exception:
                error = traceback.format_exc()
                logger.error("[RECONCILE] Error occured in payout #" + str(payout.id) + " " + error)
                alert.fatal("Error occured in reconcile of payout #" + str(payout.id) + " " + error)

Reconciliation transformed transient failures into recoverable events.In practice, it became one of the most valuable reliability mechanisms in the entire platform

Final Thoughts

One lesson that extends far beyond payment systems is that whenever you are dealing with the lifecycle of an entity that moves through a series of well-defined stages, there is often an opportunity to model the problem as a state transition system

Few Examples

• Orders move through states
• Claims move through states
• Loan applications move through states
• KYC verification moves through states
• Payment disbursements move through states

Once the lifecycle is modeled explicitly, business rules become easier to reason about, failures become easier to recover from, and systems become significantly more observable

Finite State Machine can become an essential tool in engineer’s toolkit