How to Build a Reliable Webhook System in Python — Sending, Receiving, and Not Losing Events

APIs pull data. Webhooks push it. Most implementations get the push wrong.

Reliable Webhook Event Pipeline

We integrated with a payment provider that sent webhook notifications for every transaction — payment succeeded, payment failed, refund issued, dispute opened. The integration took two days to build. It took three weeks to make it reliable.

The first week, we lost 23 webhook events. Our endpoint returned 200 OK before processing finished, the database transaction failed silently, and we never retried because we’d already acknowledged the event. Twenty-three customers were charged but never received their orders.

The second week, we started processing webhooks synchronously and our endpoint took 4 seconds to respond. The payment provider’s timeout was 5 seconds. Under load, some requests took 6 seconds. The provider marked those as failed and retried. We processed the same event twice. Eleven customers got duplicate order confirmations.

The third week, we built it properly — signature verification, async processing with Celery, idempotency with event deduplication, and a dead letter queue for failed events. It’s been running without a single lost or duplicated event for over a year.

Here’s everything I learned, distilled into the patterns that would have saved me three weeks.

Receiving Webhooks: The Endpoint That Can’t Fail

Receive Fast, Process Later

A webhook receiver has one job: accept the event and return 200 as fast as possible. The actual processing happens later. This separation is the key to reliable webhook handling.

# BAD — processes synchronously, slow and fragile
@app.post("/webhooks/payments")
async def receive_payment_webhook(request: Request):
    payload = await request.json()

    # All of this happens before we respond
    order = db.query(Order).filter(Order.payment_id == payload['payment_id']).first()
    order.status = 'paid'
    db.commit()
    send_receipt_email(order)           # 2 seconds
    update_inventory(order)             # 500ms
    notify_warehouse(order)             # 300ms
    sync_to_accounting_system(order)    # 1.5 seconds

    return {"status": "ok"}  # 4.3 seconds later

The provider is waiting for your response. If you take too long, they’ll time out, mark the delivery as failed, and retry — creating duplicate processing. If any step fails, the entire webhook is lost because you haven’t stored the raw event.

# GOOD — accept fast, process later
@app.post("/webhooks/payments")
async def receive_payment_webhook(request: Request):
    body = await request.body()
    signature = request.headers.get("X-Webhook-Signature", "")

    # Step 1: Verify the signature (fast, no I/O)
    if not verify_signature(body, signature):
        raise HTTPException(status_code=401, detail="Invalid signature")

    # Step 2: Parse the payload
    payload = json.loads(body)

    # Step 3: Store the raw event immediately
    event_id = payload.get("event_id")
    event = WebhookEvent(
        event_id=event_id,
        source="payment_provider",
        event_type=payload.get("type"),
        payload=payload,
        status="received",
    )
    db.add(event)
    db.commit()

    # Step 4: Queue for async processing
    process_webhook_event.delay(event.id)

    # Step 5: Respond immediately
    return {"status": "received"}

This endpoint does four things and nothing more: verify the signature, parse the payload, store the raw event, and queue it for processing. Total response time: under 50ms. The provider gets their 200 OK. The event is safely persisted. If processing fails later, the raw event is in your database and you can retry it.

Signature Verification: Trust Nobody

Webhook Signature Verification

Webhook endpoints are public URLs. Anyone who discovers them can send fake events. Without signature verification, an attacker can send {"type": "payment.succeeded", "amount": 0} and your system processes it as a legitimate payment.

Every serious webhook provider signs their payloads — Stripe, GitHub, Shopify, Twilio. The signature proves the event came from them, not from an attacker.

Stripe’s Pattern (HMAC-SHA256):

import hmac
import hashlib

def verify_stripe_signature(payload: bytes, signature_header: str, webhook_secret: str) -> bool:
    """
    Stripe sends: t=timestamp,v1=signature
    We reconstruct the signed payload and compare.
    """
    try:
        elements = dict(
            item.split("=", 1) for item in signature_header.split(",")
        )
        timestamp = elements["t"]
        expected_sig = elements["v1"]

        # Stripe signs: timestamp + "." + raw body
        signed_payload = f"{timestamp}.{payload.decode()}"
        computed_sig = hmac.new(
            webhook_secret.encode(),
            signed_payload.encode(),
            hashlib.sha256
        ).hexdigest()

        return hmac.compare_digest(computed_sig, expected_sig)
    except Exception:
        return False

Generic HMAC Verification (Works for Most Providers):

def verify_webhook_signature(
    payload: bytes,
    signature: str,
    secret: str,
    algorithm: str = "sha256"
) -> bool:
    """Generic HMAC signature verification."""
    hash_func = getattr(hashlib, algorithm)
    computed = hmac.new(
        secret.encode(),
        payload,
        hash_func
    ).hexdigest()

    return hmac.compare_digest(computed, signature)

The hmac.compare_digest function is critical — it performs a constant-time comparison that prevents timing attacks. A regular == comparison short-circuits on the first different character, leaking information about the expected signature through response timing.

Always verify on the raw bytes, not the parsed JSON. JSON parsing can reorder keys, change whitespace, or modify number formatting. The signature was computed on the exact bytes the provider sent. Parse after verification, not before.

Idempotency: Process Each Event Exactly Once

Webhook Event Deduplication

Webhook providers retry failed deliveries. Your endpoint might receive the same event 2, 3, or 10 times. Without idempotency, you process it every time — sending duplicate emails, charging customers twice, or creating duplicate records.

# The WebhookEvent model tracks what we've seen
class WebhookEvent(Base):
    __tablename__ = "webhook_events"

    id = Column(Integer, primary_key=True)
    event_id = Column(String, unique=True, index=True)  # Provider's event ID
    source = Column(String, nullable=False)
    event_type = Column(String, nullable=False)
    payload = Column(JSON, nullable=False)
    status = Column(String, default="received")  # received, processing, completed, failed
    attempts = Column(Integer, default=0)
    created_at = Column(DateTime, default=func.now())
    processed_at = Column(DateTime, nullable=True)
    error_message = Column(Text, nullable=True)

# In your webhook endpoint — check before storing
@app.post("/webhooks/payments")
async def receive_payment_webhook(request: Request):
    body = await request.body()
    signature = request.headers.get("X-Webhook-Signature", "")

    if not verify_signature(body, signature):
        raise HTTPException(status_code=401)

    payload = json.loads(body)
    event_id = payload["event_id"]

    # Check if we've already received this event
    existing = db.query(WebhookEvent).filter(
        WebhookEvent.event_id == event_id
    ).first()

    if existing:
        # Already received — acknowledge but don't reprocess
        return {"status": "already_received"}

    event = WebhookEvent(
        event_id=event_id,
        source="payment_provider",
        event_type=payload["type"],
        payload=payload,
    )
    db.add(event)
    db.commit()

    process_webhook_event.delay(event.id)
    return {"status": "received"}

The event_id with a unique constraint is the deduplication key. If the provider sends the same event twice, the second attempt hits already_received and returns 200 without reprocessing. The provider stops retrying. No duplicates.

Async Processing: The Celery Task

Webhook Processing State Machine

The actual event processing happens in a Celery task — separate from the HTTP request, with retries, error handling, and status tracking:

# tasks/webhook_tasks.py
from celery import shared_task
from datetime import datetime, timezone
import logging

logger = logging.getLogger("webhooks")

@shared_task(bind=True, max_retries=5, default_retry_delay=60)
def process_webhook_event(self, event_id: int):
    event = db.query(WebhookEvent).get(event_id)

    if not event:
        logger.error(f"WebhookEvent {event_id} not found")
        return

    if event.status == "completed":
        logger.info(f"Event {event.event_id} already completed, skipping")
        return

    event.status = "processing"
    event.attempts += 1
    db.commit()

    try:
        # Route to the appropriate handler
        handler = get_event_handler(event.event_type)
        if handler:
            handler(event.payload)
        else:
            logger.warning(f"No handler for event type: {event.event_type}")

        event.status = "completed"
        event.processed_at = datetime.now(timezone.utc)
        db.commit()

        logger.info(f"Processed webhook event: {event.event_id}")

    except RetryableError as exc:
        event.status = "failed"
        event.error_message = str(exc)
        db.commit()

        countdown = 60 * (2 ** self.request.retries)  # Exponential backoff
        logger.warning(
            f"Retryable error on {event.event_id}, "
            f"attempt {self.request.retries + 1}/5, "
            f"retrying in {countdown}s: {exc}"
        )
        raise self.retry(exc=exc, countdown=countdown)

    except PermanentError as exc:
        event.status = "failed"
        event.error_message = str(exc)
        db.commit()

        logger.error(
            f"Permanent failure on {event.event_id}: {exc}",
            exc_info=True
        )
        # Don't retry — this error won't resolve itself

def get_event_handler(event_type: str):
    """Route events to their handlers."""
    handlers = {
        "payment.succeeded": handle_payment_succeeded,
        "payment.failed": handle_payment_failed,
        "refund.created": handle_refund_created,
        "dispute.opened": handle_dispute_opened,
    }
    return handlers.get(event_type)

The distinction between RetryableError and PermanentError is crucial. A network timeout calling your email service is retryable — wait 60 seconds and try again. A missing order for the given payment ID is permanent — no amount of retrying will create the order. This is the smart retry strategy from Article #25.

Sending Webhooks: The Other Side

Outbound Webhook Delivery System

If your API needs to notify other systems of events — “order created,” “user signed up,” “payment processed” — you’re building a webhook sender. This is harder than receiving because you’re responsible for delivery guarantees.

# models/webhook_subscription.py
class WebhookSubscription(Base):
    __tablename__ = "webhook_subscriptions"

    id = Column(Integer, primary_key=True)
    url = Column(String, nullable=False)
    secret = Column(String, nullable=False)  # For signing payloads
    events = Column(JSON, default=list)       # ["order.created", "order.shipped"]
    is_active = Column(Boolean, default=True)
    failure_count = Column(Integer, default=0)
    created_at = Column(DateTime, default=func.now())

class WebhookDelivery(Base):
    __tablename__ = "webhook_deliveries"

    id = Column(Integer, primary_key=True)
    subscription_id = Column(Integer, ForeignKey("webhook_subscriptions.id"))
    event_type = Column(String, nullable=False)
    payload = Column(JSON, nullable=False)
    status = Column(String, default="pending")  # pending, delivered, failed
    response_code = Column(Integer, nullable=True)
    response_body = Column(Text, nullable=True)
    attempts = Column(Integer, default=0)
    next_retry_at = Column(DateTime, nullable=True)
    created_at = Column(DateTime, default=func.now())

# services/webhook_sender.py
import hmac
import hashlib
import httpx
import json
from datetime import datetime, timezone

class WebhookSender:
    def __init__(self):
        self.client = httpx.Client(timeout=10.0)

    def dispatch_event(self, event_type: str, data: dict, db: Session):
        """Send an event to all subscribed endpoints."""
        subscriptions = (
            db.query(WebhookSubscription)
            .filter(
                WebhookSubscription.is_active == True,
                WebhookSubscription.events.contains([event_type])
            )
            .all()
        )

        for sub in subscriptions:
            delivery = WebhookDelivery(
                subscription_id=sub.id,
                event_type=event_type,
                payload=data,
            )
            db.add(delivery)
            db.commit()

            deliver_webhook.delay(delivery.id)

    def sign_payload(self, payload: bytes, secret: str) -> str:
        """Generate HMAC-SHA256 signature."""
        return hmac.new(
            secret.encode(),
            payload,
            hashlib.sha256
        ).hexdigest()

# tasks/webhook_delivery.py
@shared_task(bind=True, max_retries=8)
def deliver_webhook(self, delivery_id: int):
    delivery = db.query(WebhookDelivery).get(delivery_id)
    subscription = delivery.subscription

    payload_bytes = json.dumps(delivery.payload).encode()
    signature = WebhookSender().sign_payload(
        payload_bytes, subscription.secret
    )

    timestamp = str(int(datetime.now(timezone.utc).timestamp()))

    try:
        response = httpx.post(
            subscription.url,
            content=payload_bytes,
            headers={
                "Content-Type": "application/json",
                "X-Webhook-Signature": f"t={timestamp},v1={signature}",
                "X-Webhook-Event": delivery.event_type,
                "X-Webhook-Delivery-ID": str(delivery.id),
            },
            timeout=10.0
        )

        delivery.response_code = response.status_code
        delivery.response_body = response.text[:1000]
        delivery.attempts += 1

        if 200 <= response.status_code < 300:
            delivery.status = "delivered"
            subscription.failure_count = 0
        else:
            raise RetryableError(f"Endpoint returned {response.status_code}")

    except (httpx.TimeoutException, httpx.ConnectError, RetryableError) as exc:
        delivery.attempts += 1
        delivery.status = "failed"
        subscription.failure_count += 1

        # Disable subscription after 15 consecutive failures
        if subscription.failure_count >= 15:
            subscription.is_active = False
            logger.warning(
                f"Disabled webhook subscription {subscription.id} "
                f"after {subscription.failure_count} consecutive failures"
            )

        db.commit()

        # Exponential backoff: 1m, 2m, 4m, 8m, 16m, 32m, 64m, 128m
        countdown = 60 * (2 ** self.request.retries)
        raise self.retry(exc=exc, countdown=countdown)

    finally:
        db.commit()

Key design decisions:

Automatic disable after 15 failures. If a subscriber’s endpoint is permanently down, you don’t keep retrying forever. After 15 consecutive failures, the subscription is deactivated. You can notify the subscriber via email and let them re-enable it when their endpoint is fixed.

Exponential backoff up to ~2 hours. Eight retries with doubling delays: 1 minute, 2 minutes, 4 minutes… up to about 2 hours. This gives the receiver plenty of time to recover from temporary outages without hammering them.

Store every delivery attempt. The WebhookDelivery table is your audit trail. When a subscriber says "we never received the event," you can show them the delivery log — the response code, the response body, and the number of attempts.

Webhook Delivery Retry Strategy

Sign every payload. Your subscribers need to verify that the webhook came from you, not from an attacker who discovered their endpoint URL.

The Admin Dashboard

Give your team visibility into webhook health:

@app.get("/admin/webhooks/stats")
def webhook_stats(db: Session = Depends(get_db)):
    total = db.query(WebhookEvent).count()
    completed = db.query(WebhookEvent).filter(
        WebhookEvent.status == "completed"
    ).count()
    failed = db.query(WebhookEvent).filter(
        WebhookEvent.status == "failed"
    ).count()
    pending = db.query(WebhookEvent).filter(
        WebhookEvent.status.in_(["received", "processing"])
    ).count()

    return {
        "total_events": total,
        "completed": completed,
        "failed": failed,
        "pending": pending,
        "success_rate": f"{(completed / total * 100):.1f}%" if total else "N/A",
    }

A success rate below 99% means something needs attention. Failed events need investigation. Pending events older than 10 minutes are probably stuck.

Webhook Admin Health Dashboard

Bottom Line

Webhooks look simple — receive a POST request, process it, return 200. But the gap between “works in development” and “works in production” is enormous. Events get lost, duplicated, delivered out of order, and processed after timeout. Every one of these failure modes requires a specific pattern to handle.

The architecture that works: receive fast (verify signature, store raw event, return 200), process later (Celery task with retries), deduplicate always (unique event ID), and track everything (delivery table as audit trail). It’s the same receive-store-process pattern used by Stripe, GitHub, and every other platform that handles millions of webhook events daily.

Those 23 lost events in our first week? They would have been caught by the store-before-process pattern. Those 11 duplicates in our second week? They would have been prevented by the event ID deduplication. The patterns aren’t complex. The cost of not having them is.

What’s the hardest webhook integration you’ve built? I’m curious about the edge cases — out-of-order events, payload format changes, rate limits from the provider. The community always has stories I haven’t heard. Share yours in the comments.

Thanks for reading! ♥️

A special thanks to **Level Up Coding** for giving writers and engineers a space to share practical, real-world lessons like this. I’m grateful for the opportunity to publish this piece with the publication and contribute to a community that cares about better engineering.

If this helped you, consider clapping (50 👏s), following, or sharing it. A writer without readers is just talking to themselves—so your time means everything.

Let’s keep building better together.

— Anas Issath