The Guardian Pattern: Selective Event Buffering for Ordered Retry in Distributed Systems

Samuel Ballesteros · June 2026

Most Kafka retry patterns have a blind spot. This post is about that blind spot, why it matters at production scale, and a pattern I’ve designed to fix it.

Event enters the consumer, guardian checks gates, no gates yer, event is processed.

processing fails, guardian creates a gate for further events that depend on this one which is sent to retry.

Next event is processed, and guardian finds that a prior event on which this one depends on left a gate, so this event being processed is sent to the buffer until the one that it depends on succeeds or completely fails.

The Problem

Modern event-driven systems built on Kafka rely on strict event ordering for correctness. In domains like order management, payments, or inventory, events form lifecycle chains — order_created must succeed before order_updated can be safely processed, which must succeed before order_cancelled can run. Each step depends on the one before it.

When a consumer fails to process an event, you can’t leave it blocked in its partition — that stops every event behind it, a problem known as head-of-line blocking. The standard solution is to move failing events to a retry topic and reprocess them with exponential backoff, letting the main consumer keep moving.

But this creates a new problem: subsequent events for the same entity start arriving while the failing event is still retrying. If you process them out of order, you corrupt state. So they need to be held somewhere until the failing event resolves.

Existing patterns handle this by maintaining a lock on the entity: when an event fails, every subsequent event for that entity gets blocked and redirected into the retry flow. When the failing event resolves, a tombstone signal releases them.

This works — but it’s too coarse. It locks the entire entity, not just the events that actually depend on the failing one.

Consider an order management system. If order_created fails and is retrying, there's a good reason to block order_updated — it can't run without a created order. But what about fraud_score_updated? That event is produced by an independent risk engine. It references the same order ID, but it has no dependency on whether the order creation succeeded. In a financial context, delaying a fraud score update carries real risk. Yet under existing patterns, it gets blocked anyway.

This is the first gap.

The second gap is what happens when the failing event never recovers. When an event exhausts its retries and lands in the dead letter queue, existing solutions have no structured mechanism to decide: which buffered events should be discarded, and which should be released? Without that answer, the only available move is to evict everything — including events that had nothing to do with the failed one.

This is the selective eviction problem, and to our knowledge no existing pattern addresses it.

The Core Idea

The Guardian Pattern introduces a typed failure gate mechanism built on one key insight: dependencies between event types are known at design time and can be declared statically.

Instead of locking an entire entity when something fails, you declare exactly which event types are affected by the failure of each event type. This declaration is a configuration artifact called the Guardian Rules:

{
  "order_created":              { "on_failure_buffers": ["order_updated", "order_cancelled"] },
  "order_updated":              { "on_failure_buffers": ["order_cancelled"] },
  "order_cancelled":            { "on_failure_buffers": [] },
  "customer_notification_sent": { "on_failure_buffers": [] },
  "analytics_event_tracked":    { "on_failure_buffers": [] },
  "fraud_score_updated":        { "on_failure_buffers": [] }
}

When order_created fails, only order_updated and order_cancelled get buffered. fraud_score_updated flows freely — because it's not in the list, there is no code path that can ever buffer it. This is not a runtime check; it is a structural consequence of the data model.

Two Categories of Events

The Guardian Rules encode a distinction that existing patterns ignore: not all events for an entity are equally dependent on each other.

Lifecycle events form a strict dependency chain. Each one requires the successful completion of the previous one. In the order example: order_created → order_updated → order_cancelled.

Isolated events reference the same entity but carry no dependency on the lifecycle state. fraud_score_updated, analytics_event_tracked, customer_notification_sent — these are produced by systems that operate independently.

Every event type belongs to exactly one category, fixed at declaration time. Isolated events carry an empty on_failure_buffers list, which means no gate can ever reference them, which means they can never be buffered. The protection is architectural, not operational.

The Gate

When a lifecycle event fails and is sent to the retry topic, the Guardian creates a gate — a lightweight record in Redis:

blocking_event_id:   'evt_abc123'      → unique ID of the failed event instance
blocking_event_type: 'order_created'   → used to look up the Guardian Rules
entity_id:           'order_1'         → the affected entity
buffers_types:       ['order_updated', 'order_cancelled']

The gate is keyed by the unique event ID of the specific failing instance — not just the event type. This is the design decision that makes selective eviction possible.

Every subsequent event that arrives for this entity and whose type appears in buffers_types is written to the buffer with a buffered_by field pointing to evt_abc123. This field records not just that the event is waiting, but which specific gate instance is responsible for its fate.

A gate has three possible outcomes:

• Released — the retrying event succeeds. The gate is removed. All entries where buffered_by = evt_abc123 are retrieved, sorted by their original production timestamp, and re-injected into the retry topic in order.
• Evicted — the retrying event exhausts its maximum retries. All entries where buffered_by = evt_abc123 are permanently discarded. Everything else is untouched.
• Superseded — a second gate becomes active for the same entity because a different lifecycle event also fails. Both gates exist independently and resolve independently.

Why the buffered_by Field Solves Selective Eviction

Without buffered_by, the buffer only tells you that an event is waiting. You can't tell why it's waiting, or which failing event it's tied to.

When the failing event lands in the dead letter queue, the only query available to you is: “discard everything buffered for this entity.” That destroys events that were waiting for a completely different gate — or events that were waiting for a gate that already resolved successfully.

With buffered_by, the eviction query becomes exact:

evict all buffer entries where:
  entity_id   = the affected entity
  buffered_by = the permanently failed event ID

This query touches nothing else. Two gates for the same entity are as independent as gates for two different entities.

How It Flows

Every event arriving at the main consumer passes through the Guardian before any business logic runs:

1. The Guardian checks whether any active gate for this entity lists the incoming event’s type as a dependent. This is a single Redis lookup — O(1).
2. If the event is blocked, it’s written to the buffer with buffered_by set to the gate's event ID, and the Kafka offset is committed. The consumer never stalls.
3. If the event is not blocked, it’s processed normally.
4. If processing fails, a gate is created and the event is forwarded to the retry topic.

On the retry consumer:

1. The event is reprocessed.
2. If it succeeds, the gate is released: buffered entries are retrieved, sorted by production timestamp, and re-injected into the retry topic in order. The gate is deleted atomically with this operation.
3. If it fails but hasn’t exhausted retries, it goes back to the retry topic with the next backoff delay. The gate stays active.
4. If it has exhausted retries, the gate is evicted: all owned buffer entries are discarded, an eviction notification is emitted for each, and the event is forwarded to the dead letter queue.

Every path through both consumers ends with a Kafka offset commit. The partition never stalls.

Implementation Notes

Both the gate store and the buffer store live in Redis. The Guardian Rules are loaded into memory at consumer startup.

Gate and buffer keys are structured for direct lookup:

gate:{entity_id}:{blocking_event_id}
buffer:{entity_id}:{event_id}

Gate release and eviction must be atomic. Redis MULTI/EXEC transactions cover the retrieval, deletion, and re-injection steps. If Redis becomes unavailable, the recommended behavior is to stop the consumer rather than proceed — processing an event that should have been buffered can produce an invalid state transition that is difficult to reverse.

Crash recovery is natural: both stores survive consumer crashes. If a crash occurs between writing a buffer entry and committing the Kafka offset, Kafka re-delivers the event. Because the buffer key includes the event ID, writing the same event twice is idempotent. This guarantee requires that event IDs are assigned by the producer and carried in the payload or headers — not generated by the consumer at processing time.

Observability

The Guardian emits six signals you should monitor:

• gate.created — a new gate is active; track blocking chains
• gate.released — successful retry resolution
• gate.evicted — permanent failure; alert on this
• buffer.entry.added — buffer write; watch for growth
• buffer.entry.released — buffered event re-injected after gate release
• buffer.entry.evicted — buffered event discarded; part of the dead letter audit trail

A buffer that grows without corresponding gate releases is an early sign that a gate has stalled. Monitor gate age against the maximum retry window to catch this before it becomes a production incident.

Correctness Properties

A correct implementation satisfies five properties, all of which hold by construction rather than by runtime enforcement:

Isolation preservation. An isolated event is never buffered. Because isolated events have an empty on_failure_buffers list, no gate can ever list their type. The check cannot return true for them. No guard clause needed.

Selective eviction. When a gate is evicted, only its owned entries are discarded. The buffered_by field makes this an exact equality query, not an inference problem.

Ordered release. Buffered events are re-injected sorted by their original Kafka production timestamp — the business sequence in which they were produced, not the order they arrived at the consumer.

Gate independence. Two gates for the same entity are resolved independently. The resolution of one gate touches only the entries it owns.

Durability. Both stores survive consumer crashes. State is fully recoverable without data loss.

What Existing Patterns Get Wrong

The sequential retry pattern operates at entity granularity. When an entity is locked, every event for that entity is blocked — no distinction between dependent and isolated events. When permanent failure occurs, the tombstone mechanism was designed for the success case only; there is no structured signal to separate dependent events from independent ones. The only option is to evict everything.

The queue ladder approach focuses on a different problem: ensuring reliable reprocessing through progressive backoff. It doesn’t model dependencies between event types at all. The two approaches are complementary — the Guardian Pattern sits in front of the retry infrastructure regardless of how that infrastructure is organized internally.

Three Rules

The Guardian Pattern reduces to three rules:

Rule 1 — Declare dependencies statically. Every event type declares which event types its failure will affect, once, at configuration time. A developer reading the rules for any event type sees its full downstream impact without tracing a graph.

Rule 2 — Tag every buffered event with its owner. Every buffer entry carries the unique ID of the specific event instance that caused it to be buffered. Not the event type — the specific instance.

Rule 3 — Resolve gates independently. The resolution of one gate never affects the state of any other gate, even for the same entity.

Adoption

The Guardian Pattern is designed to be adopted incrementally. It doesn’t replace your retry infrastructure — it sits in front of it as a thin decision layer. The migration from an entity-level blocking pattern requires three changes: add the Guardian Rules configuration, introduce the buffered_by field in the buffer store, and replace the tombstone signaling mechanism with direct gate release calls from the retry consumer.

What’s Next

Open questions worth exploring: empirical measurement of buffer growth under high-failure-rate conditions; formal verification of the atomicity requirements under Redis failure scenarios; and dynamic Guardian Rules — configurations that can be updated at runtime without consumer restart, for systems where the event dependency structure evolves frequently.

The Guardian Pattern was originally conceived and formally proposed by Samuel Ballesteros (June 2026).