Designing a Production-Grade Webhook System: Lessons from Payments, Reliability, and Scale

When people hear the word webhook, they often think of it as just an HTTP callback. Something happens in one system, and another system gets notified. Simple.

But in real production systems, especially in domains like payments, refunds, order management, logistics, and user account workflows, a webhook is much more than a callback. It becomes a reliability boundary, a security boundary, and sometimes even the difference between a successful customer journey and a production incident.

A webhook system is an event-driven mechanism where one application automatically notifies another application when a specific event occurs. Instead of continuously polling to check whether something has changed, the source system pushes the event to a registered callback URL.

A simple analogy is a doorbell. You do not keep opening the door every few minutes to check whether someone is outside. The visitor rings the bell, and you get notified instantly. That is exactly what a webhook does in software systems.

For example, in a payment flow, when a customer completes a transaction, the payment gateway may send a webhook to the merchant system saying that the payment was successful, failed, refunded, or disputed. The merchant system can then update the order status, trigger shipment, notify OMS, or send a confirmation to the customer.

But once we move from a simple example to a real production-grade webhook system, many serious design questions appear.

Start with Requirements, Not Technology

Before jumping into Kafka, queues, workers, retries, or dashboards, the first step is to clarify requirements.

A strong system design discussion should start with questions like:

What type of events are triggering webhooks? Are they critical events like payment confirmation, refund success, order creation, or account activation? Or are they non-critical events like analytics, logs, or monitoring samples?

This distinction changes the architecture completely. A payment success event cannot be treated the same way as a page-view analytics event. If a payment webhook is lost, the customer may be charged but the order may not be confirmed. If an analytics event is lost, the business may lose one data point, but the customer journey is not broken.

For critical flows, we need retries, persistence, dead-letter queues, idempotency, monitoring, and acknowledgements. For non-critical flows, we may prefer fast ingestion, batching, async processing, and lightweight consumers.

This is where senior engineers stand out. They do not just pick the most popular tool. They understand the business impact first and then design accordingly.

Functional Requirements of a Webhook System

At a functional level, a webhook system should allow clients to register, update, and delete webhook subscriptions. A client should be able to provide a callback URL and say, “Send me these specific events here.”

For example, a client may register:

callbackUrl = https://client.com/payment/webhook
eventTypes = PAYMENT_SUCCESS, PAYMENT_FAILED

The system should expose APIs such as:

POST /webhooks
PATCH /webhooks/{id}
DELETE /webhooks/{id}
GET /webhooks/{id}

Behind the scenes, this webhook configuration must be stored in a reliable metadata store. This configuration usually contains the client ID, callback URL, subscribed event types, status, retry policy, rate limit, signing secret, created timestamp, and updated timestamp.

The system should also support event filtering. Not every client wants every event. One client may only care about payment success and failure, while another may care about refunds and chargebacks. This means the webhook system must efficiently route events only to subscribed clients.

At small scale, a simple database lookup may be enough. At high scale, we need indexing, caching, partitioning, and sometimes an internal publish-subscribe pattern to avoid scanning all subscriptions for every incoming event.

Non-Functional Requirements Matter Even More

Functional requirements define what the system does. Non-functional requirements define how well it does it.

For a production-grade webhook system, the key non-functional requirements are scalability, high availability, fault tolerance, security, durability, and observability.

If the system is expected to handle one billion events per day, that translates to around 11,500 events per second on average. Peak traffic can easily go beyond 100,000 events per second. A single server or monolithic setup will not survive that load. We need distributed ingestion, horizontally scalable workers, partitioning, rate limiting, and backpressure management.

High availability is also critical because webhook systems are time-sensitive. If the webhook pipeline is down, clients may not receive payment confirmations, order updates, or refund statuses. The system should be deployed across multiple availability zones, use load balancers, health checks, redundant queues, replicated databases, and multiple worker instances.

Fault tolerance is another major requirement. Failures are normal in webhook systems. Callback URLs may be down. Clients may return 500 errors. Networks may time out. DNS may fail. Workers may crash. The system must recover gracefully instead of silently dropping events.

Delivery Guarantees: Best Effort vs At-Least-Once vs Exactly-Once

One of the most important design discussions in webhook systems is delivery guarantee.

Best effort delivery means the system will try to deliver the event, but if it fails, data loss is acceptable. This works for logs, analytics, metrics, and monitoring samples. It is simple, fast, and low-cost, but data loss is possible.

At-least-once delivery means the system guarantees that the event will be delivered at least once, but duplicates are possible. This is the most common model in real-world webhook systems. If the client does not acknowledge the event, the system retries. The trade-off is that the same event may be delivered multiple times.

This is why idempotency is mandatory.

For example, if the same payment success webhook arrives twice, the receiver should check:

Has this paymentId already been processed?

If yes, the duplicate event should be ignored.

Most payment gateways and webhook providers follow this model because it is practical, scalable, and reliable enough for production systems.

Exactly-once delivery sounds ideal because it means no duplicates and no missing events. But in distributed systems, this is extremely hard. Network failures, uncertain acknowledgements, database commits, retries, and consumer crashes make exactly-once semantics operationally expensive and complex.

In most real-world webhook systems, the practical choice is:

At-least-once delivery + idempotent consumers

That gives a strong balance between reliability and operational simplicity.

Event Ordering: Does Sequence Matter?

Another important question is whether event ordering matters.

Some systems can process events independently. Others cannot.

In a payment system, event order can be critical. Consider this lifecycle:

PAYMENT_CREATED
PAYMENT_AUTHORIZED
PAYMENT_CAPTURED
REFUND_INITIATED

If REFUND_INITIATED is processed before PAYMENT_CAPTURED, the system may try to refund a payment that has not yet been successfully captured. That creates an invalid state.

The common solution is partitioning. If all events related to the same payment ID go to the same Kafka partition, Kafka can preserve ordering within that partition.

For example:

Partition Key = paymentId

This ensures all events for payment P100 go to the same partition and are processed sequentially.

But this comes with a trade-off. Ordering reduces parallelism. If one partition becomes too hot, throughput may suffer. Therefore, in payment systems, we usually need per-payment ordering, not global ordering across all payments.

Security Is Not Just “Use HTTPS”

Security is one of the most overlooked areas in webhook design discussions. But for payment systems, order systems, account systems, and financial workflows, it is critical.

Webhook endpoints are often public. Anyone can try to send a fake request. If the system blindly accepts it, an attacker could send a fake payment success event or replay an old valid event.

A secure webhook system should enforce HTTPS for encryption in transit. But HTTPS alone is not enough.

The system should also use HMAC signature validation. In this model, the sender and receiver share a secret. The sender signs the payload using that secret, and the receiver recomputes the signature using the same payload and same secret. If both signatures match, the request is considered authentic and untampered.

This protects against payload tampering.

Replay prevention is also important. An attacker may capture a valid webhook and send it again later. To prevent this, webhooks should include timestamps, unique event IDs, or nonce values. The receiver can reject old requests and ignore duplicate event IDs.

For sensitive industries, payload-level encryption may also be needed. In such cases, the client may provide a public key during webhook registration, and the webhook system can encrypt the payload so that only the client can decrypt it using its private key.

A strong webhook security design should cover authenticity, integrity, confidentiality, and replay protection.

Fault Tolerance Is More Than Retries

Many candidates say, “We will retry with exponential backoff,” and stop there. But that is not enough for a production-grade webhook system.

Fault tolerance requires a complete recovery strategy.

The system should interpret HTTP status codes intelligently. A 200 OK means successful delivery. A 400 Bad Request or 404 Not Found usually indicates a client-side issue and should not be retried aggressively. A 500 Internal Server Error, 503 Service Unavailable, or timeout usually indicates a transient failure and should be retried using exponential backoff.

Every event should be persisted before being queued for delivery. This ensures that even if the worker crashes or the queue is temporarily unavailable, the original event is not lost.

A dead-letter queue is also essential. If an event fails all retry attempts, it should be moved to a DLQ for investigation, replay, or manual recovery.

The system should also track delivery attempts. For each attempt, we should know the webhook ID, event ID, attempt number, response code, error message, latency, and timestamp.

This is the difference between a system that merely “tries again” and a system that can actually explain and recover from failures.

Scalability: Isolate Blast Radius

Scalability is not just about adding more servers. It is about isolating blast radius.

A hot client may suddenly receive a huge number of events. For example, a large merchant during a flash sale may receive millions of payment or order events. If we process all clients through the same shared queue, one hot client can slow down everyone else.

To avoid this, we can use per-client queue partitioning, adaptive rate limiting, priority queues, and dedicated worker pools for large clients.

Similarly, a hot source application can overwhelm ingestion. If Stripe, Shopify, or a warehouse scanner suddenly sends 10 times its normal volume, the system should detect the anomaly, alert the team, and apply throttling if necessary.

For high write throughput, the database should also be optimized using batch inserts, partitioning, proper indexing, and replication.

Real scalability means the system can handle spikes without allowing one tenant or source to degrade the entire platform.

Observability: The System Must Explain What Happened

In production, things will fail. The question is not whether failures happen. The question is whether the system can explain them.

Observability in a webhook system has two major goals.

First, track the delivery status of every webhook event. Clients and operators should be able to answer:

Did this event get delivered? Which endpoint received it? How many retries happened? What response code did the client return? Why did it fail? Can we replay it?

Second, monitor failure patterns before they become incidents. DLQ growth, retry spikes, queue lag, high callback latency, and increasing 5xx responses should trigger alerts.

A good design separates the event table from the delivery tracking table. The event table is append-only and stores the original event. The delivery table stores each delivery attempt because one event may be delivered to multiple webhooks, and each delivery has its own lifecycle.

For example, event EVT_123 may succeed for webhook A but fail for webhook B. Therefore, storing one delivery status directly on the event record would be incorrect.

Observability is the eyes and ears of the system. Without it, even a well-designed webhook platform becomes a black box.

Final Thoughts

A webhook system may look simple from the outside, but production-grade webhook design requires careful thinking across reliability, security, scalability, durability, ordering, idempotency, and observability.

The strongest architecture is not the one with the most technologies. It is the one that matches the actual business requirements.

For non-critical events, lightweight best-effort delivery may be enough. For payment and order systems, we need durable storage, retries, idempotency, DLQs, monitoring, signature validation, replay protection, and clear recovery paths.

In interviews, do not present webhook design as a memorized architecture. Treat it as a collaborative discussion. Clarify requirements, explain trade-offs, and justify decisions based on the use case.

That is what separates a good system design answer from a production-ready engineering discussion.