Production Authorization: The Dual-Write Hazard, and a Blueprint for Machine Identity

Hardening a relationship-based authorization system against its worst failure mode, and designing the path to authenticate machines as well as humans.

Authorization, the Google Way, Part 3: The Machines & Production. Part 1 built relationship-based authorization with Ory Keto. Part 2 moved authentication to an identity-aware proxy. This final article hardens the system for production and lays out the machine-identity design. Code: **github.com/sylvester-francis/folio**.

Across this series I’ve built an authorization system I’d defend in a design review: a Zanzibar-style relationship model in Ory Keto (Part 1), and an identity-aware proxy that authenticates at the edge (Part 2). Every test is green. The demo is clean.

Photo by Zulfugar Karimov on Unsplash

And there is still a bug in the create() handler that no unit test will catch, because it is not a logic error. It is a distributed-systems error, surfacing only when one of two backing systems fails at precisely the wrong instant. This article is about that class of problem: the requests that have no human behind them, and the production failure modes that the happy path conceals. The dual-write hazard and the 403-versus-404 behavior are in the code today; the machine-identity layer is the designed next increment, and I'll be explicit about which is which.

The Challenge: The Demo Hides the Hard Parts

A working authorization demo answers the easy questions. Production asks the hard ones, and they fall into two buckets.

Identity beyond humans. Everything so far assumed a person: a browser, a cookie, a session. But production traffic is full of callers that are not people, namely batch jobs, webhooks, sibling microservices, and increasingly AI agents. None of them have a cookie, and issuing them synthetic human sessions is exactly the wrong instinct.

Consistency under failure. The moment authorization data lives in a system separate from your application data (and with a dedicated authorization service like Keto, it always does), you have two systems that do not share a transaction. Every write that touches both opens a window where one can succeed and the other can fail. The demo never sees that window. Production lives in it.

This article addresses both: how to model machine identity so the existing authorization layer is reused unchanged, and how to handle the multi-system write that can otherwise corrupt your permission state.

The Vision: One Authorization Model for People and Machines

The goal is that adding non-human callers should not require a second authorization system. A relationship-based model makes this achievable, because a “subject” was never specifically “a user.” It is an entity in a graph, identified by a string.

Various forms of interaction (Human vs Machine)

Humans authenticate through Kratos and carry sessions. Machines authenticate through Ory Hydra, the OAuth2 and OpenID Connect server, using the client-credentials grant: a service presents a client_id and client_secret, receives an access token, and sends it as a bearer token. The authorization layer behind both is identical.

Mental model. Kratos is passport control for people; Hydra is the badge office for robots. Different desks, different credentials, but once inside the building the door locks read an ID string and do not care which desk issued it.

Architecture: Adding Machine Identity Without Touching Authorization (the design)

This layer is not yet in the repository; it is the designed next increment, and the point of this section is that the design is deliberately small. You add Hydra to the stack, issue client credentials per service, and add one authenticator to Oathkeeper, the oauth2_introspection handler, alongside the cookie_session handler from Part 2.

Request flow

A request now authenticates as either a human session or a service token. Critically, the application’s authorization code does not change. PermissionGuard calls can(subjectId, namespace, permit, object), and Keto evaluates a relation tuple. To Keto, the subject id is just a string; it does not distinguish a person from a service account. The full permit hierarchy, group expansion, and folder inheritance from Part 1 apply unchanged to machine subjects.

Architectural payoff. This is the dividend of modeling authorization as relationships rather than roles. Because subjects are graph nodes identified by strings, extending the system to machines is an authentication change, not an authorization change. The most security-critical layer stays exactly as it was, already tested, already proven.

The Core Hazard: The Dual-Write Problem

Here is the most important production caveat in the entire project. Consider create():

async create(ownerId: string, input: CreateDocumentInput): Promise<FolioDocument> {
  const doc = { id: randomUUID(), ...input, ownerId, createdAt: now, updatedAt: now }

  this.store.set(doc.id, doc)                    // write #1: the document store
  await this.permissions.grantUser(              // write #2: Keto (ownership)
    ownerId, Namespace.Document, Relation.Owners, doc.id,
  )
  return doc
}

There are two writes to two independent systems with no transaction spanning them. The application database does not know about Keto, and Keto does not know about the database. Between those two await points lies a failure window:

The inverse failure is equally bad: write the grant first and let the row fail to persist, and you have a dangling tuple, an ownership grant referencing a document that does not exist.

The reason this is so dangerous is that it is invisible to your test suite. Unit and integration tests exercise the happy path where both writes succeed; they do not crash the process between two awaits. The defect lives entirely in the failure window. It passes review, passes CI, and works in the demo, then manifests weeks later when a pod is terminated mid-request and a user is permanently locked out of their own data.

Mental model. You are shipping a house in two envelopes: the deed (the database row) and the keys (the Keto ownership tuple). If only one arrives, the result is useless: a deed to a house you cannot enter, or keys to a house that is not yours. The two systems make no promise to deliver together.

Defusing It

Because there is no distributed transaction across the datastore and Keto, you need an explicit reconciliation strategy. Two production-grade options:

Production grade paths

What must never ship is the naive two-await sequence with no compensation. Folio deliberately ships exactly that, annotated with a comment marking the seam, so the hazard is visible before you choose how to close it. A tutorial is allowed to expose a sharp edge; a production system has to file it down.

A Deliberate Behavior: 403, Not 404, on a Deleted Resource

A subtle behavior falls out of the guard ordering from Part 1, and it is a security property worth understanding. When an owner deletes a document, the service purges its tuples from Keto. A later request for that id behaves like this:

Response from delete route

Because the permission check runs before the handler, and the tuples are gone, the guard returns 403, not 404. This is the correct, defensible choice:

A system that distinguishes “forbidden” (403) from “gone” (404) leaks the existence of every id a caller is not permitted to see, enabling enumeration. By returning 403 for both, Folio refuses to disclose existence. It gets this property for free from running authorization before existence, with no special-case code.

Trade-off. If your product needs friendly 404s for resources a user could access but that are genuinely gone, you must check existence and permission together in the handler and accept the existence disclosure. That is a legitimate decision; make it deliberately rather than by accident.

A Production Decision: Opaque Sessions vs. JWTs

Kratos issues opaque session cookies, validated server-side per request via whoami. The alternative is a signed JWT verified locally with no network call. The trade is real:

Mental model. An opaque session is a coat-check ticket: the cloakroom holds the authority and can refuse a stolen ticket instantly. A JWT is a festival wristband: nobody phones home, so it keeps working until it expires, even after you’ve been ejected.

The operational corollary: do not cache session lookups in the application without an invalidation story. A cached session outlives a logout, and a permission system that honors revoked sessions is not a permission system. When a hot downstream hop genuinely cannot afford the round-trip, let Oathkeeper mint a short-lived JWT at the edge (its id_token mutator) while keeping the revocable Kratos session as the source of truth, with a deliberately short lifetime.

Real-World Impact and Next Increments

The system now authenticates humans and machines through one authorization model and has an explicit posture on its hardest failure mode. From here, each extension is localized:

• MFA and passkeys. Kratos ships TOTP and WebAuthn. The session carries an authenticator_assurance_level (aal1, aal2); gating sensitive permits such as share and delete behind aal2 is a one-check extension of the session guard, since the AAL is already on the session object.
• Nested groups. Flat membership avoids a self-referential-type error (TS2502) in the OPL typecheck. The correct fix is an explicit interface annotation to break the self-reference, never weakening the typecheck that catches malformed models.
• Multi-tenancy. Keto namespaces are global; tenancy emerges from object naming or a per-tenant root folder, with the existing inheritance traversal doing the scoping. The check call is unchanged.
• Managed Ory. The same code runs against Ory’s managed Network by changing only the four base URLs, the payoff of routing every Ory call through a single typed seam.

Key Takeaways for Fellow Engineers

1. Model subjects as graph nodes, not users. When identity is a string in a relationship graph, adding machine callers is an authentication change, not an authorization rewrite. People and services share one model.
2. Treat every multi-system write as a hazard. Without a distributed transaction across your datastore and your authorization service, a partial failure corrupts state. Use an ordering guarantee or an outbox; never ship the naive two-write sequence.
3. Failure-mode bugs are invisible to happy-path tests. The dual-write defect passes CI and the demo. Design for the failure window explicitly, because your tests will not.
4. Returning 403 instead of 404 is an information-disclosure decision. Running authorization before existence yields a no-leak API for free; diverging from it is a deliberate trade.
5. Opaque sessions trade latency for revocability; do not cache them away. A cached or JWT-only session that outlives a logout is a silent security regression.

Getting Started

git clone https://github.com/sylvester-francis/folio
cd folio

# Run the full suite (66 tests, 97.56% mutation score)
bun run test

# Find the dual-write seam, deliberately left visible:
#   src/documents/documents.service.ts  →  create()

The repository’s complete guide walks the machine-identity integration, the production caveats, and the full end-to-end request lifecycle in detail.

Further Reading

• Ory Hydra documentation, the OAuth2/OIDC server for machine identity (the project repo cites OpenAI among its users)
• RFC 6749, Section 4.4, the OAuth2 client-credentials grant for service-to-service calls
• Chris Richardson, Transactional Outbox pattern, a robust fix for the dual-write hazard
• AuthZed, ZedTokens, Zookies, and consistency, the deep-dive on preventing the new-enemy problem
• Zanzibar: Google’s Consistent, Global Authorization System

Join the Journey

This concludes Authorization, the Google Way: a relationship-based model that scales with product complexity (Part 1), an edge architecture that scales across services (Part 2), and a production posture that survives real failure (Part 3). Folio is open source and built to be read.

Repository: github.com/sylvester-francis/folio

If this series changed how you think about permissions, the best next step is to clone it, run the tests, and find the dual-write comment yourself. Then go read the Zanzibar paper. The good ideas are sitting in papers most teams never open.