The LLM Is Not Your Product. The Celery Worker That Calls It Is.

We were adding an AI feature to an existing pipeline — nothing glamorous, a text summarisation job. The first question our team asked was: which model do we use? GPT-4? Claude? Gemini? We spent a week on that question.

It was the wrong question.

The model turned out to be the easy part. What took us three more weeks was everything around it — the queue that fed the job, the worker that made the API call, the retry strategy for when the upstream rate limit hit us at 3am, and the fallback path for when we needed to route tickets even if the inference job never completed. That infrastructure is what we actually shipped.

The model is a third-party API. Treat it like one.

We were using Celery for our background jobs already. Python on the application side, Celery workers handling the heavier async processing. When we added the LLM call, we dropped it into a worker the same way we’d drop in a call to Twilio or Stripe. HTTP client, credentials from AWS Secrets Manager, response parsing, error handling.

That framing changed everything. Nobody asks “which payment processor architecture should we use?” They ask “what happens when the payment API times out?” That’s the right question for LLMs too.

Rate limiting is the first thing that will bite you.

LLM APIs have two limits: requests per minute and tokens per minute. Ours was a background job-intensive system, and we hit the token limit before the request limit — something we hadn’t anticipated. A ticket with a long thread costs five times the tokens of a short one.

We landed on a token-estimation step before enqueue. We estimate the token count from the input length, then slot the job into a rate-limited queue with a sliding window counter in Redis. Expensive tickets go to a separate lower-throughput queue. Same pattern we’d use for any API with tiered limits.

Timeouts and fallbacks are not optional.

LLM inference can be slow — 10, 15, 20 seconds for a large context. In a synchronous web request, that’s a non-starter. In an async worker, it’s still a problem: jobs pile up, queues bloat, and when the model provider has an incident your worker pool stalls.

We set aggressive timeouts on the HTTP client — 12 seconds — with a dead-letter queue for jobs that fail after three retries. The fallback behavior for a failed summarisation is that the ticket routes without a summary. That decision — define the fallback before you write the happy path — is the same discipline you need for any external dependency.

The teams shipping reliable AI features are backend teams.

Not because they know more about models. Because they treated the model like any other upstream service: designed for failure, built the queue first, wrote the retry logic before the feature logic.

The model is a commodity. Every month there’s a new one that’s cheaper or faster. The Celery worker that calls it, handles the failure, and keeps your system moving when the API is down — that’s the thing you built. That’s the product.

The teams that are still debugging flaky AI features in production are the ones who spent their first week picking a model.