Photo by Planet Volumes on Unsplash

Timeouts and Error Taxonomy: What Happens If Gemini Hangs?

FastAPI + Gemini chatbot series — Part 7 (Final)

I’ve locked the door, limited the speed, budgeted the output. What remains is the project’s sneakiest danger: an external service like Gemini can be slow to respond, can hang, or can throw an error. If I set no limit, my connection could wait forever; if I catch the error wrong, the whole session crashes. This final part answers two questions: how do you put a timeout on a streaming call, and how do you separate Gemini’s different error types without dropping the user?

wait_for: the simplest timeout

asyncio.wait_for(aw, timeout) bounds a "waitable" job to a set duration. If it finishes in time, you get the normal result; if the time runs out, it cancels the job and raises TimeoutError. The official example:

async def eternity():
    await asyncio.sleep(3600)   # sleeps an hour
    print("yay!")

async def main():
    try:
        await asyncio.wait_for(eternity(), timeout=1.0)
    except asyncio.TimeoutError:
        print("timeout!")

In my notes I wrote two critical points:

1. Always catch TimeoutError. If you don't wrap it in try/except asyncio.TimeoutError, the app blows up with this error when the time runs out — just like a server error.
2. The work is actually cancelled. wait_for doesn't merely "stop waiting"; it also cancels the underlying coroutine. So a hung Gemini call doesn't keep running for nothing — it stops.

The streaming complication: wait_for can't wrap a loop

Simple so far. But my Gemini call isn’t a single await, it's a stream: I receive the reply chunk by chunk. And wait_for can only wrap a single awaitable — it can't wrap an async for loop. Splitting the call into two stages makes this clear:

# Stage 1: a single await — sends the request to Gemini, sets up the "pipe", returns the stream object
stream = await chat.send_message_stream(data)

# Stage 2: the loop — reads the chunks coming from that pipe one by one, until the reply ends
async for chunk in stream:
    ...

wait_for can wrap only Stage 1:

stream = await asyncio.wait_for(chat.send_message_stream(data), timeout=30)
async for chunk in stream:           # Stage 2 now flows freely
    ...

So is that enough? The reasoning in my notes: Gemini hanging usually happens at Stage 1 — i.e., I sent the request but it never started replying. Once the reply starts (Stage 2), chunks flow steadily. So “bounding only the start” covers most cases. But what if I wanted to catch a hang mid-stream, in Stage 2? wait_for wouldn't do; I'd need another tool.

asyncio.timeout + reschedule: from total timeout to idle timeout

asyncio.timeout(delay) (Python 3.11+) returns a context manager that binds an entire block to a deadline, rather than a single call like wait_for. So it can wrap the whole async for loop:

loop = asyncio.get_running_loop()
async with asyncio.timeout(IDLE_TIMEOUT) as cm:
    async for chunk in await chat.send_message_stream(data.strip()):
        if chunk.text:
            await websocket.send_text(chunk.text)
        cm.reschedule(loop.time() + IDLE_TIMEOUT)

The heart is the last line. The conceptual distinction here is this part’s most important lesson:

Total timeout: “The entire reply must finish in at most 30 seconds.” — Wrongly cuts off a long but healthy reply. Idle timeout: “At most 30 seconds may pass without a new chunk.” — Triggers only if the stream stalls.

For LLM streaming the right one is idle timeout. A model can stream chunks nonstop for two minutes — that’s not a problem, just a long reply. What I don’t want is the chunks stopping and the connection hanging. cm.reschedule(...) turns the total timeout into an idle timeout by pushing the deadline 30 seconds forward each time a chunk arrives.

This distinction wasn’t theoretical for me — it’s the actual history of my code. My first working version was a flat async with asyncio.timeout(10): with no reschedule: a total timeout. It passed every quick test, because short replies finish well under ten seconds. The flaw only shows when you imagine a long, perfectly healthy reply — the timeout would cut it off mid-sentence for the crime of being thorough. That realization is what pushed me from "bound the whole block" to "bound the silence," and the old version still sits commented out in my main.py as a fossil of the wrong first idea.

Two supporting details:

• loop = asyncio.get_running_loop() is needed because reschedule wants an absolute time, not a relative one. To say "now + 30 seconds" I need a clock that gives me "now": the event loop's own clock, loop.time().
• cm.reschedule(loop.time() + IDLE_TIMEOUT) → "move the deadline to 30 seconds from now."

When the time runs out (i.e., no new chunk for 30 seconds), the block raises TimeoutError; I catch it, send the user a polite message, and continue back to the top of the loop:

except asyncio.TimeoutError:
    ws_logger.warning("Gemini reply stream stalled (idle timeout)")
    await websocket.send_text("Please try again.")
    continue

continue is again critical: the timeout drops a single message, not the whole connection. I chose 30 seconds as a reasonable starting point; it can be tuned.

A practical testing trick worth sharing: how do you prove a timeout path works when Gemini normally answers in two seconds? You make the timeout absurd. While developing, I temporarily set the limit to 0.0001 seconds — guaranteeing the deadline fires before any real chunk could arrive — and confirmed the except asyncio.TimeoutError branch actually ran: the warning hit the log, the user message went out, the loop continued. Then I put the real value back. An error path you've never seen fire is an error path you're merely hoping works.

Gemini error taxonomy: ServerError and ClientError

A timeout is one kind of failure; but Gemini can also raise explicit errors. The SDK’s error hierarchy is clean: the base class is APIError, with two children — ClientError (HTTP 4xx, something wrong in your request) and ServerError (HTTP 5xx, something wrong on Google's side). In both, e.code gives the HTTP status and e.message the message. I use this distinction directly in my code:

except ServerError as e:
    if e.code == 503:
        ws_logger.warning("Google API 503 — overloaded")
        await websocket.send_text("Service is busy")
    else:
        ws_logger.error(f"Google API Server Error ({e.code}): {e}")
        await websocket.send_text("Server error")
    continue

except ClientError as e:
    if e.code == 429:
        ws_logger.warning("Google API 429 — quota exceeded")
        await websocket.send_text("My quota is full")
    else:
        ws_logger.error(f"Google API Client Error ({e.code}): {e}")
        await websocket.send_text("I couldn't understand your request.")
    continue

This structure encodes three separate decisions:

• Whose fault? A ServerError is temporary and not my fault (Google is busy/down); a ClientError is a problem in my request (quota exhausted, invalid parameter).
• Log level. 503 (busy) and 429 (quota) are expected, temporary conditions → WARNING. Other codes are genuinely unexpected → ERROR, with the full {e} detail (because I'll need to investigate them). This continues the "two channels" principle from Part 5: detail to the log, a neutral message to the user.
• What to tell the user. A clear, separate sentence per case: “Service is busy”, “My quota is full”, “I couldn’t understand your request.”

And again every block ends with continue: even if one message hits a Gemini error, the connection doesn't close, and the user can keep going. These except blocks are inside the message loop; their scope is "one message."

Putting it all together: the full stream block

try:
    loop = asyncio.get_running_loop()
    async with asyncio.timeout(IDLE_TIMEOUT) as cm:
        async for chunk in await chat.send_message_stream(data.strip()):
            if chunk.text:
                await websocket.send_text(chunk.text)
            cm.reschedule(loop.time() + IDLE_TIMEOUT)

except asyncio.TimeoutError:
    ws_logger.warning("Gemini reply stream stalled (idle timeout)")
    await websocket.send_text("Please try again.")
    continue

except ServerError as e:
    if e.code == 503:
        ws_logger.warning("Google API 503 — overloaded")
        await websocket.send_text("Service is busy")
    else:
        ws_logger.error(f"Google API Server Error ({e.code}): {e}")
        await websocket.send_text("Server error")
    continue

except ClientError as e:
    if e.code == 429:
        ws_logger.warning("Google API 429 — quota exceeded")
        await websocket.send_text("My quota is full")
    else:
        ws_logger.error(f"Google API Client Error ({e.code}): {e}")
        await websocket.send_text("I couldn't understand your request.")
    continue

This block lives inside the lifecycle from Part 4 (WebSocketDisconnect / general Exception / finally). So there's a hierarchy: outermost is the connection itself (disconnect, unexpected crash, cleanup), and inside it each message's own failure modes (timeout, server, client). Each layer handles the error within its own scope.

Where I started vs. where I ended

At the start: “I found wait_for for timeouts but couldn't figure out how to apply it to streaming — it couldn't wrap the loop. I was also handling Gemini errors with a single catch-all except."

At the end: “wait_for wraps a single awaitable, not a loop; a streaming call has two stages (set up the pipe / read the pipe). To catch a mid-stream stall too, I turned a total timeout into an idle timeout with the asyncio.timeout context manager + reschedule — because a long reply isn't a problem, a stall is. I split Gemini errors into ServerError/ClientError, gave each its own log level and user message, and kept them all within single-message scope via continue."

Closing the series

Over seven posts, starting from async programming’s promise to “not waste time while waiting,” we reached a production-ready AI chatbot:

1. Sync / Async — why the project’s infrastructure is async end to end.
2. FastAPI First Steps — the server, OpenAPI, path operations, decorators.
3. Typed Data / Pydantic — automatic conversion and validation via type hints.
4. WebSocket Lifecycle — the connection’s three phases, disconnect vs. server error, close codes, finally.
5. Config & Logging — fail-fast with pydantic-settings, field_validator, getLogger(__name__), LoggerAdapter.
6. Security & Cost — auth, origin, input validation, per-user rate limiting, GenerateContentConfig.
7. Timeouts & Error Taxonomy — wait_for vs. asyncio.timeout, idle timeout, ServerError/ClientError.

The real takeaway wasn’t a single chatbot; it was learning to ask, on every line, “how does this behave when I’m not watching? What if someone abuses it? What if the external service hangs? What if the library shifts under me?”

A note on where this series sits: I’m following a six-cycle roadmap for this project, and these seven posts cover its first three cycles plus the timeout half of cycle 4. What remains on the map — and what the next round of posts would cover — is: the structured message envelope and chat-history windowing (the rest of cycle 4, so the frontend can tell “chunk” from “done” from “error”, and so token costs stop growing cumulatively), splitting the single file into layers with an LLM abstraction and lifespan (cycle 5), and tests, metrics, and graceful shutdown (cycle 6). Deliberately deferred, deliberately listed.

Open questions

• I made the timeout per-chunk (idle); in some cases a total cap (e.g., “a single reply at most 5 minutes”) might also be wanted?
• When a timeout/disconnect happens mid-stream, where exactly does billing stop on Gemini’s side? Per tokens produced, or end to end?
• On a 429 (quota), is telling the user "my quota is full" enough, or should I build a back-off (retry/backoff) strategy?

References 📖

• Python — asyncio.wait_for
• Python — asyncio.timeout and Timeout.reschedule
• Google Gen AI SDK — Error handling (APIError, ClientError, ServerError)
• RFC 6455 — WebSocket close codes