Python FastAPI Lifespan Shutdown Dropped In-Flight Celery Publishes Mid-Deploy

SIGTERM killed the app before send_task finished. Jobs never existed.

Rolling deploy Wednesday 14:00. Celery queue depth flat. orders table showed 890 rows status=enqueue_pending — stuck between DB commit and broker publish.

Customers saw confirmed orders. Warehouse saw nothing to pick. 14 hours until someone traced SIGTERM timing to missing send_task.

FastAPI lifespan shutdown was 3 seconds. Celery Redis publish during traffic drain took longer. Pod gone — message never enqueued.

This is deploy mechanics, not Celery bugs. Kubernetes sends SIGTERM. Your app must drain or persist. We did neither for the publish gap.

The stuck rows query that found it

SELECT count(*), min(created_at), max(created_at)
FROM orders
WHERE status = 'enqueue_pending';

890 rows. Oldest 14 hours. created_at clustered in 5-minute windows matching deploy times from CI dashboard. Smoking gun.

The lifespan we shipped

@asynccontextmanager
async def lifespan(app: FastAPI):
    await init_db_pool()
    yield
    await close_db_pool()  # only cleanup we implemented

app = FastAPI(lifespan=lifespan)

Order creation path:

@app.post("/orders")
async def create_order(body: OrderCreate):
    order_id = str(uuid4())
    async with db.transaction():
        await db.insert_order(order_id, body, status="enqueue_pending")
    celery_app.send_task("fulfillment.process", args=[order_id])
    await db.update_order_status(order_id, "queued")
    return {"order_id": order_id}

Race on shutdown:

1. Pod receives SIGTERM — Kubernetes starts drain
2. Request in flight — DB insert committed
3. send_task blocked on Redis connection or TCP slow
4. Uvicorn shutdown timeout — 3s default in our chart
5. Process exit — no task, row stuck enqueue_pending

No Celery task ID. No retry. Monitoring on queue depth missed it — task never arrived.

**30 Production Incidents That Cost $10K+** — deploy-correlated silent loss shows up in the case files with dollar estimates and the reconciliation queries ops wishes they’d run hourly.

Graceful shutdown tuning — necessary, not sufficient

Uvicorn --timeout-graceful-shutdown — we set 30 after incident.

# deployment.yaml
lifecycle:
  preStop:
    exec:
      command: ["sh", "-c", "sleep 5"]
terminationGracePeriodSeconds: 60

Buys time. Does not fix transactional outbox.

We also had BackgroundTasks for post-response publish — same class of bug. Lifespan shutdown does not wait for background tasks unless configured. Two anti-patterns, one deploy window.

Trend opinion — shutdown is part of the API contract

Most teams design startup — pools, migrations, cache warm. Shutdown is “hope SIGKILL is late enough.”

With Kubernetes, SIGTERM is guaranteed. Design for it.

Three patterns ranked:

1. Transactional outbox (what we run now):

async def create_order(body: OrderCreate):
    order_id = str(uuid4())
    async with db.transaction():
        await db.insert_order(order_id, body, status="pending")
        await db.insert_outbox(
            event_type="fulfillment.process",
            payload={"order_id": order_id},
        )
    return {"order_id": order_id}

# separate publisher loop — at-least-once to broker
async def outbox_publisher():
    while not shutdown_event.is_set():
        rows = await db.fetch_pending_outbox(limit=50)
        for row in rows:
            celery_app.send_task(row.event_type, args=[row.payload])
            await db.mark_outbox_sent(row.id)
        await asyncio.sleep(0.5)

Pod dies mid-request — outbox row not committed or fully published with marker. Publisher retries.

2. Confirm broker ACK before HTTP 200:

result = celery_app.send_task(..., ignore_result=False)
result.get(timeout=5)  # wait broker accept — blocks response, fragile

We rejected — latency and still not end-to-end exactly-once.

3. Reaper for stuck rows:

SELECT order_id FROM orders
WHERE status = 'enqueue_pending'
  AND created_at < now() - interval '5 minutes';

Re-enqueue from cron. Backstop, not architecture.

Lifespan shutdown done properly

shutdown_event = asyncio.Event()

@asynccontextmanager
async def lifespan(app: FastAPI):
    publisher_task = asyncio.create_task(outbox_publisher())
    yield
    shutdown_event.set()
    await asyncio.wait_for(publisher_task, timeout=25.0)
    await close_db_pool()

Graceful shutdown waits for publisher drain up to 25s. Matches K8s terminationGracePeriodSeconds: 60 minus preStop sleep.

Log on exit:

pending = await db.count_outbox_pending()
if pending:
    logger.warning("shutdown_with_pending_outbox", extra={"count": pending})

Detection we added

1. Metric — orders_enqueue_pending_age_seconds histogram
2. Alert — any row enqueue_pending > 2 minutes
3. Deploy annotation — compare pending spike to rollout

increase(orders_enqueue_pending_total[5m]) > 10

Celery publish reliability notes

1. broker_transport_options — visibility timeout for Redis
2. task_publish_retry — retries on connection errors
3. Publisher confirms — RabbitMQ; Redis has different semantics

Outbox abstracts broker flakiness from HTTP handler.

Kubernetes endpoint drain — what stops first

On SIGTERM, Service endpoints remove pod before full grace in some configurations. In-flight requests still run — but new requests stop. Our stuck publishes were in-flight during drain — not new traffic. Rollout strategy maxUnavailable: 1 limited blast radius but did not eliminate it.

Readiness probe must reflect “accepting work” vs “shutting down”:

@app.get("/ready")
async def ready():
    if shutdown_event.is_set():
        raise HTTPException(503, "draining")
    return {"ok": True}

Endpoint drops pod from rotation immediately on shutdown signal — stops new orders during drain.

Honest limits

Outbox adds table, publisher, and idempotent consumers. More moving parts. Moving parts beat lost orders.

terminationGracePeriodSeconds is not infinite — long shutdown blocks deploy velocity. Size grace to p99 publish drain, not worst case forever.

If you use FastAPI lifespan without shutdown logic, you are betting workers finish before kube gets impatient. We lost that bet 890 times in one deploy window.

Warehouse impact — why stuck rows hurt operations

Each enqueue_pending order passed fraud and reserved inventory. Reservations expired after 24 hours — ~200 SKUs returned late. Picking team idle Wednesday morning while support insisted orders showed confirmed.

Reconciliation re-enqueued 847 of 890. 43 needed manual fix — customers had re-ordered. Outbox shipped Thursday. Two deploys per week since — zero gap.

Testing shutdown in CI

We added integration test — start app, fire request with slow Redis mock, send SIGTERM mid-handler, assert outbox row exists and eventually publishes:

def test_shutdown_preserves_outbox(redis_mock, client):
    redis_mock.slow_publish = True
    r = client.post("/orders", json=payload)
    os.kill(app_pid, signal.SIGTERM)
    wait_for(lambda: db.count_outbox_pending() == 0, timeout=30)

Flaky twice. Worth it. Catches lifespan regressions.

Celery send_task retry on publish failure

def publish_with_retry(task_name, args, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return celery_app.send_task(task_name, args=args)
        except kombu.exceptions.OperationalError:
            if attempt == max_attempts - 1:
                raise
            time.sleep(0.5 * (attempt + 1))

Helps broker blip — not pod SIGTERM mid-publish. Outbox still required. Publish retry is belt; outbox is suspenders.

Order status API lie

Customers polled GET /orders/{id} — status confirmed because DB row committed before publish failed. UI lied with database truth. Fixed status machine:

pending_payment → confirmed → fulfillment_queued → picking

confirmed only after outbox marks sent. HTTP 201 returns earlier with status: processing — mobile updated copy.

Comparison — shutdown strategies we evaluated

Longer grace only — cheap, insufficient for publish gap. Outbox — durable, more code, correct. Sync publish before response — blocks latency, still loses on SIGKILL mid-write. Kafka transaction — overkill for our volume.

Pick outbox for money paths. Pick longer grace as supplement, not substitute.

BackgroundTasks overlap — same deploy, same lesson

During lifespan fix sprint we found three routes still using BackgroundTasks for post-order emails. Same SIGTERM vulnerability — smaller blast radius because email loss is recoverable. Migrated to outbox anyway. One pattern, one fix.

rg "BackgroundTasks|send_task" app/ --glob "*.py"

Draw sequence diagram: DB commit → publish → response. Mark SIGTERM window. If window exists without outbox, you have the bug.

Lifespan shutdown is not boilerplate you copy from tutorial part one and skip part two. Part two is where orders disappear.

Uvicorn vs Gunicorn graceful timeout alignment

Chart had terminationGracePeriodSeconds: 60. Uvicorn default graceful 15s inside Gunicorn master. Master killed workers before Kubernetes killed pod — race with in-flight publishes. Set both explicitly in same PR:

# gunicorn_conf.py
graceful_timeout = 45
timeout = 120

Document the relationship in runbook. On-call should not discover it during deploy.

Warehouse SLA missed Wednesday ship window — executive visibility finally got shutdown on roadmap. Pain drives priority. Outbox should not have waited for pain.

If you use FastAPI lifespan without shutdown logic, you are betting workers finish before kube gets impatient. We lost that bet 890 times in one deploy window.

Reaper cron still runs — backstop for outbox publisher crash. Belt and suspenders. Money paths deserve both.

Deploy is not instant. Your shutdown handler should know that.

SIGTERM is not a suggestion. It is a deadline. Design outbox, grace periods, and readiness drains around that hard fact. No exceptions on payment paths. Fulfillment and billing deserve the same durability bar.

Three actions this week:

1. Read Helm terminationGracePeriodSeconds vs Uvicorn graceful timeout
2. Grep send_task after DB writes — draw failure window
3. Query stuck intermediate statuses — define reaper or outbox

Deploy is not instant. Your shutdown handler should know that.

Froquiz Senior Dev Challenge scenarios cover deploy failure, message loss, and Python service design — practice for the outage you have not had yet.

→ **Froquiz**

Before you go

• Please take a moment to like the post and follow the writer!
• Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute here