Framework-First — My experience architecting an async framework

Notes on pulling services out of a monolith at a startup, the framework I built underneath them, and two outages I earned the hard way.

Photo by Fotis Fotopoulos on Unsplash

The monolith had been around for the better part of our journey when we started carving services out of it. It wasn’t bad code, exactly. It was code that everyone had quietly stopped trusting. You’d change one thing, and something three modules away would break in a way nobody predicted, and after that happens to you a couple of times, you start padding every estimate with “…and a day to find what I broke.”

So we started pulling pieces out. But the part I want to talk about isn’t any one service. It’s the thing I built underneath them, before I shipped a single endpoint, because I’d seen this movie before and knew how it ended if I didn’t.

Here’s how it ends. You extract service number one. Fine. You extract number two a month later, slightly differently, because you learned a few things the first time. Number three gets written by someone else entirely and authenticates requests in a way that’s almost like the other two, except it returns a different error body. Now nobody can hold the whole thing in their head, and somewhere there’s a frontend dev with a Slack message that just says “wait, which 401 does this one send?”

I didn’t want that. So before the first service went out, I spent two weeks on a framework nobody had asked me for.

This was a while ago, before any of us had an AI in the editor. No Copilot finishing a controller for you, no Cursor scaffolding a service, nothing to prompt. Every abstraction in here was a decision somebody made on purpose, usually me, usually after talking myself out of two worse ones first. I mention it because it changes how you read the rest. None of this was generated. It was thought about, slowly, by someone who’d already been burned a few times.

The boring part that matters: controllers, services, stores

Every service got the same three layers, and I was annoying about it, because consistency was the whole point.

A controller does HTTP and nothing else. Parse the request, call a service, return the response. If you find business logic in a controller, that’s a bug.

A service is where the actual decisions live. It has no idea it’s sitting behind HTTP. You could call it from a test, a CLI, a queue worker, and it wouldn’t know the difference. This is the layer I actually cared about.

A store talks to Postgres. It doesn’t know why it’s fetching a row. It just fetches it.

People have names for this — hexagonal, ports and adapters, the dependency rule. We mostly didn’t use them. We just said “controllers stay thin” enough times that it became muscle memory. The practical payoff is that your interesting code sits in the middle, bored and safe, while the stuff that actually changes (HTTP at one edge, SQL at the other) stays out where you can mess with it without a cold sweat.

Adding a new domain is genuinely boring, which is the nicest thing I can say about it. A folder, three files, fill in the parts that are actually about your problem. The rest is handled a layer down. This was exactly what I needed the framework to do. “Make the right choice, the default one”.

One decorator does the annoying stuff

The glue is a decorator. Every endpoint is a static method on a controller, and every one of them wears @ServiceManager.register(...). You pass it arguments describing what the endpoint needs, and it deals with all of that before your function runs.

class OrderController(BaseController):
    @staticmethod
    @ServiceManager.register(
        body_schema=CreateOrderSchema,
        response_schema=OrderSchema,
        atomic=True,
        allowed_roles=(UserRole.ADMIN,),
        audit=True,
        audit_action=AuditAction.CREATE_ORDER,
    )
    async def create_order(ctx: RequestContext):
        order = await OrderService.create(ctx.auth, ctx.body)
        return ApiResponse(data=order)

By the time create_order runs, the caller is authenticated, their role is checked, the body has been parsed and validated against CreateOrderSchema, and a transaction is open. So the handler is four lines, and all four of them are about orders. That's the pitch.

I should be honest about the inside of register, though, because the flattering version of this post would pretend it's elegant. It is not. It opens with a wall of asserts checking that you didn't pass a response schema without subclassing the right base, that you didn't switch on auditing without giving it an identifier, and so on. There's a comment in the middle that just says # do not reorder. The @staticmethod has to go above the register decorator or route discovery silently skips the method, and the endpoint just doesn't exist, no error, which cost me an afternoon exactly once. It's a workhorse, and it looks like one.

And not every endpoint is this tidy. There’s one method in the billing controller pushing 300 lines because the proration rules have about a dozen special cases, and every time someone “cleans it up” they reintroduce a bug we already fixed. So it stays. Some code earns the right to be ugly.

Pointing a URL at a controller is one line, in one routes file:

RouteRegistry.register(path="/orders", controller=OrderController)

The registry inspects the controller, works out which HTTP verbs it implements, and supports a v1/v2 split so we can evolve an endpoint without a flag day. The controller has no idea any of that is happening, which is exactly how I wanted it. We also have some ways to map non-HTTP verb methods to it.

Async, because we’re mostly just waiting

The whole thing is async top to bottom — Starlette in front, SQLAlchemy over asyncpg behind. The reason is unglamorous. A service like this spends nearly all of its wall-clock time waiting on something else: Postgres, an internal service, Stripe. If you’re going to wait anyway, you may as well wait on a few thousand requests at once with a handful of workers instead of burning a whole thread every time something blocks.

The one rule I policed in review was that async is all-or-nothing down a call path. One blocking requests.get somebody pasted in from Stack Overflow, and the event loop freezes for every request that the worker is currently juggling. Making the base classes async by default meant you had to go out of your way to get it wrong, and those are the only rules that survive contact with a deadline.

Errors look the same everywhere

Small thing, big payoff. There’s a little family of exceptions (BadRequest, AccessDenied, ConflictError, a few more) and one piece of middleware that turns any of them into the same JSON envelope. A service raises ConflictError("subscription already active") and stops thinking about it. The middleware sets the status code, logs it the same way every time, and returns a body that looks like every other error in the system. Anything it doesn't recognise becomes a plain 500 with the full trace in our logs and nothing useful leaking to the caller.

The win showed up on the frontend. They wrote their error handling once. By the time we shipped the fourth service, their code already knew what its errors looked like, because everything in the building speaks the same dialect.

The transaction you can’t forget

atomic=True up in that decorator wraps your handler in a db.transaction(). Commit if it returns, roll back if it throws. I leaned on this harder than anything else, and billing is the reason.

One “create subscription” might write to five tables: the subscription row, an invoice, an add-on mapping, a snapshot of the prior state, and an audit record. If the fourth write fails and the first three don’t unwind, you’ve got a customer in a state that can’t exist — an invoice attached to a subscription that never finished being created. Without the framework, staying safe means trusting every engineer who ever touches billing to remember to wrap things correctly, forever. With it, forgetting isn’t an option. You’d have to deliberately pass atomic=False and then defend that in review, which nobody ever wanted to do.

The database is sacred

If there’s one belief sitting under all of this, it’s that the application is not allowed to be the thing that guarantees your data is correct. The application has bugs. I wrote most of them. Postgres is the only layer that can actually refuse to store something wrong, so that’s where I put the real rules.

Two of those rules you’ve already seen. Only the store ever talks to the database, so every way a table can possibly be written lives in one file you can read top to bottom, instead of being smeared across a dozen services. And the transaction wrapper from earlier means a half-finished write is never a state the rest of the system can observe.

The third rule is the schema itself, and I’m genuinely paranoid about it. Almost every column is NOT NULL until something forces my hand. Every relationship is a real, enforced foreign key. Not a bare integer the team has quietly agreed points at another table and hopes stays valid. Amount columns carry CHECK constraints so a negative charge can't physically exist as a row. When a child has to belong to the same tenant as its parent, that's a composite foreign key, so the database makes a cross-tenant reference impossible instead of trusting some service to remember the check. It's a little extreme and completely deliberate: if the application hands Postgres garbage, Postgres should hand it straight back. If a bad row can't be represented in the schema, no amount of buggy code can bring one into existence.

It isn’t free. Strict constraints make a few migrations genuinely painful, and every so often a reasonable-looking write bounces off a constraint at the worst possible moment. But every time I can remember, the constraint was right and I was wrong, which is the whole argument for keeping it. I’d much rather the database tell me no today than find out next quarter that some service has been quietly writing rows that were never supposed to exist.

Two things that actually broke

The framework was the calm part. This is the part that wasn’t.

A clock that skipped an hour

Renewals ran off a daily job. Conceptually trivial: find every account whose next_billing_date is today, process it. I had it firing at 2:30 in the morning on a schedule I'd set in Eastern time, because that's the timezone I think in, which was the whole mistake in one sentence.

One morning it just didn’t run. No error, no alert, and the logs for that window were empty, which is its own special horror, because an empty log looks identical to “nothing was wrong.”

It was the second Sunday in March. Daylight saving. At 2 a.m., the clock jumped straight to 3, so 2:30 a.m. Eastern did not exist that day, and the scheduler treated a moment that doesn’t exist the way you’d expect: it did nothing. Every account due to renew that day quietly didn’t.

I found them with a query whose shape I still remember, where next_billing_date < current_date and status = 'active', then reprocessed the stragglers by hand and spent the afternoon explaining to support why a handful of customers had a billing gap. The code fix was five minutes: schedule everything in UTC and never think in local time again. The actual lesson took longer. A job that crashes pages you at 3 a.m. A job that silently doesn't run looks exactly like a job that had nothing to do. So now every scheduled job we own emits a "done, processed N" heartbeat when it finishes, and the thing we alert on is the absence of that heartbeat. We stopped letting success stay quiet.

Everyone renewing on the same morning

When we migrated accounts off the old system, the migration backfilled next_billing_date to the existing billing date, which unfortunately happened to be a single day of the month due to the existing design. I missed analysing this data before migration and how it will fit in the new system. That miss has cost me more sleep than any bug I've ever written.

Launch was fine. The following weeks were fine. Then the shared date rolled around, and the existing accounts all came due on the same morning.

The daily job grabbed the whole list and processed it the way the rest of the codebase processes lists — concurrently, no limit on how many at once. Each of those tasks reached for a database connection.. It drained our database connection pool in about a second. Everything else sharing that pool, real user traffic and the health check included, started queuing for a connection that wasn't coming back. Postgres began turning new connections away with sorry, too many clients already, which is the most ominous polite sentence in computing. The health check timed out, the orchestrator decided the instance was dead, killed it mid-batch, and started a fresh one that walked directly into the same wall.

The single thing that didn’t go wrong was the data. Every interrupted renewal was inside a transaction, so the half-finished ones rolled back clean. Small mercy — and the entire reason the framework existed in the first place.

The fix was obvious in hindsight, the way these always are. Process in batches, cap the concurrency, drop a little jitter between chunks. And during any future migration, analyse the data and how it fits in the system before migrating it. I now have a near-physical reaction to the phrase "we'll fix it next cycle."

Would I do it again

The framework? Yes, immediately, and sooner. The hard part was never the code. It was justifying two weeks on something with no demo at the end, at a startup where something is always on fire, and the fire that ships features burns the loudest. I mostly got away with it by selling it as “this is how we move faster in March,” and then March came, and we did, and nobody remembered to be annoyed about January.

What I’d actually change: I’d have built the background-job tooling (the heartbeats, the UTC defaults, the bounded concurrency) at the same time as the request framework, instead of bolting each piece on after it had already hurt me. The request path got all my upfront attention. The background path got none. Both outages came from the background path. I don’t think that’s a coincidence.

Someone asked me recently whether I’d just generate all this today, now that the tools can write a service while you blink. Honestly, some of it, yes. The boring half of register, the CRUD that basically writes itself. But the parts that earned their place came from having already been burned and remembering exactly how it felt. The tools are extraordinary at producing code. They still don't lie awake wondering whether the renewal job is idempotent. For now, that worrying is the job.

I don’t have a clean closing line for this. The framework is still running, mostly unchanged, holding up services I hadn’t imagined when I wrote it, and most days, nobody thinks about it at all. For a piece of infrastructure, that’s about the highest compliment there is.