Why Charging a Card and Updating Your Database Is Not Atomic (And How Systems Break)

What happens when your system crashes between payment and database updates — and how to design for recovery

There’s a dangerous assumption many backend systems make: that charging a customer and updating the database happen together.

They don’t. And when your process crashes in between:

$299 charged. Order stuck in PENDING. Inventory unchanged.

This isn’t a rare edge case. This is a normal failure mode. Let me show you.

The System: An AI Agent That Processes Orders

I built a system specifically to demonstrate this problem — not as a thought experiment, but as a real, breakable system.

It’s a Flask-based order processing agent powered by Claude (via the Anthropic API). The model acts as an orchestrator, calling tools to execute a multi-step workflow:

1. check_inventory — is the product available?
2. create_order — create a PENDING order in the database
3. process_payment — charge the credit card
4. reserve_and_complete — update inventory and mark the order as COMPLETED

In the happy path, everything works beautifully.

The agent checks inventory (Sony WH-1000XM5, $299, 3 in stock), creates the order, processes payment, and completes the workflow in ~9 seconds.

Clean. Elegant…And completely fragile.

Where Systems Actually Break

Let’s walk through the real failure.

Same steps. Same system.

• Inventory check
• Order created
• Payment processed (TXN-87831, $299 charged)

Then the process crashes before the final step. CRASH- process died after payment was charged! Payment taken. Stock NOT reserved. Order stuck in PENDING.

The database now tells the story:

Two orders for the same product.

Order #2 has no transaction_id — the payment happened in memory. When the process died, that information died with it.

The customer is charged. The order is stuck. Inventory is wrong.

Support gets a call.

Someone fixes it manually…maybe

You Don’t Need to Force a Crash

This isn’t artificial. In production, this happens all the time.

Infrastructure kills your process mid-flight

• Kubernetes kills your pod (OOM, deployments, node failure)
• A node dies during a rolling deployment
• The network drops between services

Your code fails silently in the gap

• Unhandled exceptions
• Schema validation errors
• SQLAlchemy deadlocks under load

Different causes. Same outcome:

Money taken. Order broken. Customer unhappy.

Why This Happens

Because this:

charge_customer()
update_database()

…is not atomic.

You can wrap your database in a transaction.

You cannot wrap a credit card charge.

Common Solutions (and Why They Hurt)

Before reaching for frameworks, this is how systems usually try to solve it.

Each approach works partially — and fails in its own way.

Database Transactions

with db.session.begin():
    product.stock -= quantity
    order.status = "COMPLETED"
    order.payment_status = "APPROVED"
    order.transaction_id = transaction_id

If anything fails, everything rolls back atomically.

Except:

Payment is an external HTTP call.

The moment you involve an external system, database atomicity stops protecting you.

Idempotency Keys

response = stripe.PaymentIntent.create(
    amount=amount,
    idempotency_key=order_id
)  # simplified example

Call the API twice with the same key, and you won’t double charge.

But:

• It only works if the provider supports it
• It doesn’t fix missed steps

You still need something that actually retries the failed operation.

Outbox Pattern

Store intent in the database and process it asynchronously.

It works.

But now you own:

• background workers
• retry logic
• dead letter queues
• monitoring

That’s easily hundreds of lines of infrastructure code.

Manual Saga Pattern

try:
    payment = process_payment(...)
    completed_steps.append(("payment_charged", payment["transaction_id"]))
    reserve_and_complete(order.id, ...)
except Exception:
    for step in reversed(completed_steps):
        if step[0] == "payment_charged":
            refund_payment(step[1])

Each step has a compensating action.

Elegant in theory.

In practice:

Compensating actions can fail too.

Now you need retries for your retries.

Complexity grows fast.

The Real Problem

All these approaches share one issue:

If your process dies while handling failure — you’re back where you started.

Enter Dapr Workflow

Dapr (Distributed Application Runtime) is a runtime for building distributed applications.

https://dapr.io/

It provides building blocks like state management, pub/sub messaging, and workflows — abstracting away the infrastructure complexity of microservices systems.

In this case, we’re using Dapr Workflow — a durable execution engine that persists state, handles retries, and resumes execution after crashes.

Instead of building:

• checkpointing
• retries
• recovery

You declare the workflow.

def order_workflow(ctx: DaprWorkflowContext, order_input: dict):
    # Each step is checkpointed. If the process crashes,
    # Dapr resumes from the last completed activity.

    inventory = yield ctx.call_activity(check_inventory, input=order_input)

    order = yield ctx.call_activity(create_order, input=order_input)

    payment = yield ctx.call_activity(process_payment, input={
        "order_id": order["id"],
        "amount": inventory["price"]
    })

    yield ctx.call_activity(reserve_and_complete, input={
        "order_id": order["id"],
        "transaction_id": payment["transaction_id"]
    })

    return {"status": "COMPLETED", "order_id": order["id"]}

The runtime handles:

• state persistence
• retries
• crash recovery

Dapr does not eliminate the need for idempotency or careful API design — it ensures your workflow survives crashes.

Each activity is checkpointed.

If the process crashes after step 3, Dapr knows steps 1–3 completed.

When the process restarts, it resumes from step 4 — with full context preserved.

Same Crash. Different Outcome.

Same scenario. Same failure.

This time with Dapr Workflow.

Payment succeeds (TXN-82315).

Then:

CRASH — process died after payment was charged!

Dapr detects:

execution failed with a recoverable error and will be retried later

The process exits.

The database looks broken — just like before.

But then the process restarts.

Dapr resumes from the last checkpoint.

reserve_and_complete runs again.

After recovery

The system recovers automatically.

No manual fixes. No support tickets. No database surgery.

Why This Matters

If you’ve never seen this happen, it’s only a matter of time.

If you have, you probably still remember the incident.

Distributed systems don’t fail in obvious ways.

They fail in the gaps between steps.

AI Agents Change the Failure Model

AI agents don’t just add another step to your system. They change how execution works.

In this system, Claude isn’t just generating text — it’s acting as an orchestrator. Instead of calling functions directly, the flow looks like this:

1. The model decides which tool to call
2. The system executes the tool
3. The result is fed back into the model
4. The model decides the next step
5. This repeats until the workflow completes.

What a traditional system looks like

check_inventory()
create_order()
process_payment()
reserve_and_complete()

Execution is linear. State lives in memory. If the process crashes, you lose the call stack.

What an AI agent actually does

LLM → "call check_inventory"
Tool executes → result returned

LLM → "call create_order"
Tool executes → result returned

LLM → "call process_payment"
Tool executes → result returned

LLM → "call reserve_and_complete"
CRASH happens here

Each step is separated by a network roundtrip, model inference, tool execution, and state passing between steps.

The difference is subtle but critical: with a traditional system, you lose the call stack. With an AI agent, there is no call stack — execution is already fragmented across tool call boundaries. So what you lose is not just execution state. You lose where you were in the decision process.

What Dapr actually persists (and what it doesn’t)

Dapr Workflow persists which activity you’re on, the inputs and outputs of each step, and the overall workflow state.

It does not persist the LLM’s internal reasoning, the full conversation history with the model, or intermediate thoughts between tool calls.

Dapr checkpoints execution state — not model cognition.

Why this still works

When the process crashes and restarts, Dapr resumes from the last completed activity. No LLM “memory” is restored. No reasoning is replayed.

This works because the workflow is structured around explicit steps. Each activity is self-contained, designed to be safely retried, and driven by persisted inputs. The LLM decides what to do next — but once a step is executed, its result becomes durable state.

Recovery doesn’t depend on reconstructing the model’s reasoning. It depends on replaying the workflow from a known checkpoint.

The real takeaway

AI agents don’t make systems magically resilient. They make failure handling more important — because execution is multi-step, decisions are external, and state is distributed across boundaries.

Without durability, a crash means starting over — or worse, ending in an inconsistent state.

With a workflow engine, steps are tracked, progress is persisted, and execution can safely continue.

Not because the model remembers. But because the system does.

Final Thought

Build the happy path first.

Then break it.

Then make sure it survives.

Because in distributed systems, failure is not an edge case.

It’s the default.

The full demo (including crash scenarios and Dapr integration) is available on GitHub: https://github.com/VelmiraPetkova/ai-order-agent