From Child Processes to Kubernetes Pods: What Actually Changes in a Deterministic AI Runtime

The execution protocol stays the same. The failure model, readiness model, lifecycle boundaries, networking, and operational guarantees do not.

The Pod was Running.

The runtime was not.Everything in this article follows from the eleven days I spent on the gap between those two sentences.

This article continues the recovery proof established in *When the Process Dies, the Execution Doesn’t: Proving Multi-Tenant AI Runtime Crash Recovery*. That first article demonstrated that killing a real external runtime process does not terminate the durable execution: in-flight DAG work resumes under the same ExecutionId, local-queued work is redispatched through its durable SharedRunId, and safe tenants remain outside the recovery path. The question in this article is what happens when the failed host is no longer a child operating-system process, but a Kubernetes Pod. The recovery contract stays the same. The host lifecycle, readiness model, routing boundary, and definition of a completed failure do not.

0. What this article is

This is an engineering post-mortem of a hosting milestone in the Deterministic AI Runtime — a multi-tenant .NET runtime for durable AI and workflow execution, with deterministic DAG execution, durable execution identities, crash recovery, replay, Redis-backed hot state, and MongoDB-backed forensic records.

The milestone: runtime instances can now be hosted as Kubernetes Pods instead of child operating-system processes, under both HTTP and gRPC transports, with the same crash-recovery guarantees.

It is not a Kubernetes tutorial. There is no kubectl apply in it. The interesting content is the set of things that turned out to be implicit in process hosting and had to become explicit before Kubernetes hosting could be trusted with durable multi-tenant work.

If you take one idea away, take this one: a distributed runtime does not have a “deployment target.” It has a host contract. Process hosting lets you get away with never writing that contract down, because in a single process every clause of it happens to be true at the same instant. Kubernetes separates those instants by seconds, and every second is a window in which the system can be confidently, observably wrong.

Three things I want to be honest about up front, because the rest of the article is more useful if you know where the edges are:

• This is validated on Minikube and Docker Desktop, not on a large production cluster.
• The highest validated concurrency is parallelism 5–15 tenants, 45 runs, ~8.6 minutes.
• Kubernetes performs exactly zero execution recovery in this design, and that is deliberate.

1. The assumption that failed

The runtime has been able to scale out to external runtime instances for a while. The flow was already durable and already tested: when admission decides that no tenant-visible capacity can accept a shared run, it writes a durable scale-out request; a watcher claims it; a provisioner asks the Host Manager for a runtime host; the host comes up, registers, publishes capacity, and normal dispatch resumes.

Until this milestone, “runtime host” meant a child operating-system process. Adding Kubernetes looked, from the top of the call stack, like swapping one line:

Process.Start(...)      ->      CreatePod(...)

That framing survived about a day.

The failure was not a crash. It was worse than a crash, because it looked like success. The Pod reported Running. kubectl get pods was green. The container had not restarted. Nothing had thrown. And the scale-out request sat there, unfulfilled, until the shared run came back with:

Status        = QueuedGlobally
FailureReason = http-endpoint-missing

The Pod was alive. The runtime was not usable.

The control plane had a runtime identity and even a capacity record, but that record was not safe to dispatch against: its externally routable transport endpoint had disappeared. And — the subtle one, the one that took longest to internalise — there was no proof that a runtime command sent through the shared route would land on that runtime instance rather than a sibling.

That last clause is where process hosting had been quietly lying to me for two years. When you start a child process on port 5031 and then connect to 127.0.0.1:5031, you are not "probably" talking to the process you started. You are definitionally talking to it. There is no routing layer, no DNS, no control-plane reconciliation loop, nothing between you and the socket. Reachability and identity are the same fact.

Kubernetes takes those two facts apart and hands them back to you separately. It also hands you back a third one you never had to think about: when they became true.

Compare what each host actually owes the control plane before dispatch is safe:

Process host                    Kubernetes host
------------                    ---------------
allocate local port             create Pod
start child process             create Service
wait for local endpoint         wait for scheduling
register runtime                wait for container start
dispatch                        wait for application start
                                resolve transport exposure
                                create or update route
                                wait for route propagation
                                verify runtime identity through the route
                                publish usable capacity
                                only then dispatch

Five steps versus eleven. The extra six are not ceremony. Each one is a distinct window in which the host is healthy and the runtime is unusable, and — this is the part that matters for a deterministic runtime — each one is a window in which the control plane can make a scheduling decision it will later regret.

An eventually consistent system can tolerate optimistic dispatch when retries are idempotent and their consequences are explicitly recorded. A deterministic runtime cannot treat those retries as invisible implementation details: every reassignment, retry, and recovery decision must remain durably classifiable, otherwise the resulting execution history cannot distinguish convergence from duplicate work.

2. The architecture boundary

The single most important decision in this milestone was refusing to let Kubernetes become a third runtime provider.

There was real pressure to do it. Kubernetes has its own health model, its own readiness probes, its own restart semantics, its own service discovery. Every one of those is a place where you can think: the platform already solves this, why am I duplicating it? And every one of those, if you accept the offer, quietly moves a piece of your execution semantics into infrastructure configuration where it cannot be unit-tested, cannot be replayed, and cannot be reasoned about by the reconcilers that own correctness.

So: Kubernetes is a Host Manager lifecycle strategy. It sits alongside ProcessAiRuntimeHostCreationStrategy as another IAiRuntimeHostCreationStrategy. It does not transport runtime commands, does not decide admission, and does not own recovery.

Shared Run
    |
    v
Admission                      decides: is there tenant-visible capacity?
    |
    v
SharedRun.Status = ScaleOutRequested
    |
    v
Redis Scale-Out Request Store  durable, claimable, replayable
    |
    v
AiRuntimeScaleOutRequestWatcherHostedService
    |
    +-------------------------------+
    |                               |
    v                               v
IAiHttpRuntimeScaleOutProvisioner   AiGrpcRuntimeScaleOutProvisioner
    |                               |
    +---------------+---------------+
                    |
                    v
           IAiRuntimeHostManager      decides: how is a host created?
                    |
       +------------+------------+
       |                         |
       v                         v
ProcessAiRuntimeHost      KubernetesAiRuntimeHost
CreationStrategy          CreationStrategy
                                 |
                                 v
                     Pod + Service + Route
                                 |
                                 v
                        Runtime readiness
                                 |
                                 v
               Registry + capacity publication
                                 |
                                 v
                Normal HTTP or gRPC dispatch

HTTP remains HTTP. gRPC remains gRPC over HTTP/2. Retry policy, timeouts, circuit-breaker state, and structured failure reasons stay in the provider where they already lived. The Kubernetes strategy returns an ordinary AiRuntimeHostStartResult, and the upstream provisioner and watcher continue owning scale-out fulfilment exactly as they do for process hosts. The normal dispatch and recovery semantics above the Host Manager do not need Kubernetes-specific branches.

The invariant I kept coming back to whenever the design pressure got high:

Kubernetes owns host lifecycle.
HTTP or gRPC owns runtime command transport.
The runtime instance owns its local queue and DAG execution.
The control plane owns registry/capacity publication and recovery coordination.

Four sentences. Every bug in this milestone was, in retrospect, a place where one of them had leaked into another. The endpoint race in section 9 is line four leaking into line three. The kubectl port-forward bug in section 11 is line one leaking into the test harness. The gRPC control-plane identity bug was line four leaking into line one.

That is not a coincidence — it is what an architecture boundary is. A boundary you can state in four sentences is a boundary whose violations you can name.

3. Why Pod Running is not runtime readiness

Process hosting lets you cheat. The host and the endpoint are the same thing. Kubernetes removes that coincidence, and readiness splits into four independent layers.

Infrastructure readiness. The Pod exists, is scheduled, the container is running, the Service exists. This is what kubectl shows you, and it is the only layer Kubernetes will ever assert on your behalf.

Application readiness. The ASP.NET host has started, the runtime instance has initialised under its assigned identity, workers are up. A container can be Running for several seconds before the process inside it has bound a socket. On a cold image pull it can be minutes.

Transport readiness. An endpoint exists, a Service or Gateway route exists, the route has propagated, and a command sent to that endpoint reaches the intended runtime. With a shared Gateway, a route that is Accepted but not yet Programmed may still yield no backend — or fall through to another configured route or default backend, depending on the controller and Gateway configuration. A successful connection alone therefore does not prove runtime identity.

Capacity readiness. A descriptor is published, CanAcceptRun is true, AvailableRunSlots is positive, endpoint metadata is present, and tenant ownership is preserved.

These cannot be collapsed into a Pod phase, because they fail independently and in both directions. A Pod can be Ready with no route. A route can be programmed to a Pod whose runtime has not finished initialising. Capacity can be published with a descriptor that is missing the endpoint — which is exactly the bug in section 9.

What made this tractable was assigning each layer an owner, not just a name:

Layer Question it answers Owner Resource creation Did the Pod/Service request succeed or converge safely? Kubernetes host client Pod readiness Is the exact Pod running and ready? Kubernetes host client Endpoint exposure Is there a unique endpoint for this runtime’s transport? Kubernetes strategy / Gateway endpoint manager Runtime command readiness Can the selected HTTP or gRPC provider reach the runtime command service? Generic runtime readiness waiter Gateway route readiness Does the shared route reach the exact selected runtime? Kubernetes strategy route probe Registry / capacity publication Is this runtime now safe to admit and dispatch work to? KubernetesAiRuntimeInstancePublisher

Six owners, six failure classes, six distinguishable log lines. Before this table existed, every one of those failures presented identically as “scale-out request stuck in Observed.”

The ordering the strategy enforces:

Pod Running          not enough
Pod Ready            not enough
Service exists       not enough
Gateway Programmed   not enough
Runtime command path works   -> now publish capacity

RequireRuntimeReadiness makes the last step mandatory. When it is on, KubernetesAiRuntimeInstancePublisher does not write a registration or a capacity descriptor until a real runtime command has completed against the resolved endpoint. Not a TCP connect. Not a health endpoint. A runtime command, answered by the runtime instance the control plane meant to reach.

The cost is startup latency. The benefit is that admission never sees a runtime it cannot use, which means the shared queue never makes a scheduling decision it has to unwind. In a deterministic system that trade is not close.

4. Exposure models, and why the tests use the ugly one

Four exposure strategies are supported: cluster Service DNS, NodePort, per-runtime kubectl port-forward, and a shared Gateway API endpoint with per-runtime routes.

They exist because “how does the control plane reach a runtime” has genuinely different answers depending on where the control plane runs. Inside the cluster, Service DNS is correct and everything else is overhead. Outside the cluster — which is the integration-test topology — Service DNS is unreachable by definition.

The integration tests use a shared Gateway plus a single local port-forward. The reasoning is arithmetic rather than aesthetic: a parallelism-5 scenario holds fifteen tenants’ worth of runtimes open simultaneously. Per-runtime port-forwarding means fifteen kubectl child processes, fifteen local ports, fifteen independent failure modes, and fifteen things to clean up when a test is cancelled. One shared Gateway data-plane endpoint routes to all of them through a single bridge.

Routing selects the target runtime by header:

x-ai-runtime-instance-id: <RuntimeInstanceId>

Control Plane
    |
    | one shared endpoint
    v
Kubernetes Gateway
    |
    | x-ai-runtime-instance-id
    +--------------------+--------------------+
    |                    |                    |
    v                    v                    v
HTTPRoute / GRPCRoute A  Route B              Route C
    |                    |                    |
    v                    v                    v
Runtime Service A      Service B            Service C
    |                    |                    |
    v                    v                    v
Runtime Pod A          Pod B                Pod C

Each runtime gets its own route resource, and the route kind follows the transport: HTTPRoute for HTTP runtimes, GRPCRoute for gRPC runtimes. These are not interchangeable. A Gateway controller that implements HTTPRoute does not necessarily implement GRPCRoute, and gRPC additionally requires that plaintext HTTP/2 survives the hop — which is why the Pod builder configures Kestrel for HTTP/2 explicitly rather than relying on protocol negotiation that a proxy may not preserve.

Worth stating because it is a real operational constraint rather than a detail: the Gateway API CRDs and a compatible controller are external prerequisites. The strategy dynamically creates GatewayClass, Gateway, and route resources. It does not install CRDs and it does not deploy a controller. A cluster without them fails at Gateway readiness with a structured error, which is the correct behaviour — but it is not self-provisioning, and any deployment guide that implies otherwise is wrong.

To be unambiguous: **kubectl port-forward is not a production ingress recommendation.* It is a local integration-test bridge for Minikube and Docker Desktop, where the control-plane process lives on the developer's machine. In a real deployment the control plane runs inside the cluster and talks to Service DNS, or through a properly provisioned Gateway. The port-forward path exists so that the same* Kubernetes lifecycle code can be exercised end-to-end on a laptop — not because it is how you should ship.

5. Kubernetes without YAML as the source of truth

Runtime Pods, Services, Gateways, and routes are created through the Kubernetes .NET SDK, not applied from manifests.

This is not an argument that YAML is bad. Static manifests are the right tool for static topology, and most of what a platform team deploys is static topology. This runtime’s runtime layer is not: a Pod exists because admission decided that a specific tenant needed capacity for a specific RuntimeInstanceId at a specific moment, and it should stop existing when that lifecycle ends. There is no meaningful desired-state file to reconcile against, because the desired state is a function of a durable queue that changes every few seconds.

Making creation part of the lifecycle call buys four concrete things.

Metadata becomes deterministic and correlated. AiKubernetesRuntimePodMetadataBuilder derives Pod names from a readable prefix plus a stable hash of the runtime identity. This solves three problems at once: long runtime identities do not blow past Kubernetes name limits; similar prefixes do not collide after truncation; and a repeated start request for the same runtime converges on the same name instead of spawning a duplicate Pod. Labels carry control-plane-id, runtime-instance-id, provider, transport, host-provider. Annotations carry the richer values that would violate label-value constraints — transport.endpoint, tenant.id, runtime.isolationMode, and the rest.

Idempotency becomes a first-class lifecycle property. Duplicate scale-out requests for the same logical runtime are not a hypothetical. They happen whenever a watcher retries, whenever a reconciler runs concurrently with an operator action, whenever recovery and normal admission both decide capacity is needed. So the strategy serialises by identity:

RuntimeInstanceId
    -> one lifecycle gate (per-runtime semaphore)
    -> one convergent start/kill sequence

A duplicate start revalidates the existing Pod and, for shared-Gateway paths, the routed command path. If the existing host is still usable, the request returns a converged result rather than creating a second Pod:

kubernetes.creation.converged        = True
kubernetes.creation.convergence.source = runtime-host-lifecycle-cache

That metadata is not decoration. When you are staring at a recovery timeline six weeks later trying to work out why there are three start events and one Pod, the difference between “created” and “converged” is the entire explanation.

Ownership becomes knowable. The create result records whether this invocation actually created the resource or adopted an existing one. That single boolean is what makes failure cleanup safe: a failed start deletes only what it created, and a converged duplicate never deletes a live Pod that another invocation owns. AlreadyExists is treated as convergence only after resource identity is validated — unrelated existing resources are never silently adopted. That rule exists because the alternative, in a shared namespace, is a runtime that deletes someone else's Pod during its own cleanup.

Tests can inspect exactly what was built. AiKubernetesRuntimePodSpec deliberately keeps SDK types out of the Host Manager contract, so the spec builder is unit-testable without a cluster while the SDK client remains the only component that needs one.

6. Runtime identity and multi-tenancy

Every Pod receives and must preserve: ControlPlaneId, RuntimeInstanceId, TenantId, TenantGroupId, provider.name, transport.name, host.provider=kubernetes, isolation mode, and the runtime instance prefix. These flow from the ExecutionContextSnapshot and the admission decision into Pod environment, labels, annotations, registry metadata, and capacity metadata — one identity, six representations, no translation layer permitted to drop a field.

The same RuntimeInstanceId must remain visible and identical through:

admission decision
  -> scale-out request
    -> host start request
      -> Kubernetes labels and annotations
        -> transport routing header
          -> registry and capacity descriptor
            -> local runtime queue
              -> recovery evidence

Eight hops. A single lossy hop anywhere in that chain produces a runtime that works perfectly right up until the moment something fails, at which point recovery cannot correlate the failure to the work.

This identity model is explained in more depth in *Why One RunId Is Not Enough: TenantId, SharedRunId, LocalRunId, and ExecutionId in a Distributed AI Runtime. The Kubernetes milestone does not replace that model; it stress-tests it. Moving the execution host across a process boundary and* a cluster boundary is the strongest argument I have found for why those identities had to be separate in the first place. A design with a single RunId cannot express "the same durable execution resumed on a different Pod," because it has no vocabulary for the difference between the work and the place the work was running.

The isolation modes are unchanged by hosting. Dedicated capacity is visible only to its owner scope. Shared capacity is visible to compatible shared scopes. Hybrid is owner-scoped but may fall back to Shared when AllowSharedFallback permits it.

The thing worth saying loudly: a namespace is not a tenant boundary. Kubernetes labels are operational evidence and a resource-selection mechanism. They are not authorization. Tenant visibility is decided by the runtime registry, the capacity store, the execution context, and the isolation evaluator — the same components that decide it for process hosts. If someone edits a label, they change what kubectl get -l returns. They do not change what a tenant can see.

Namespace strategy can complement isolation. It cannot implement it. The distinction matters the first time someone asks whether a tenant’s data can leak, because “we use separate namespaces” is an infrastructure answer to an authorization question, and it will not survive an audit.

One design choice is load-bearing here: the runtime Pod cannot be the exclusive owner of the full capacity descriptor.

The Pod can publish local runtime facts such as worker availability, queue depth, heartbeat state, and available slots. The Kubernetes host publisher owns the externally routable endpoint and route metadata, because only the control plane knows how that Pod is exposed outside its own network boundary.

That split is why capacity publication must be monotonic. A frequent local heartbeat may refresh local capacity state, but it must never erase richer control-plane metadata such as the Gateway endpoint, route identity, transport aliases, or tenant ownership.

That distinction is also the whole story of section 9: the heartbeat path knew valid local facts, but its partial view was allowed to replace a richer externally published descriptor.

7. The real crash boundary

Killing a process and deleting a Pod are not the same operation, and the difference matters for determinism rather than for tidiness.

A process kill followed by an explicit wait-for-exit gives you a synchronous, observable boundary: the process is gone, and nothing it was doing continues. DELETE /api/v1/namespaces/x/pods/y gives you an accepted request. The container may still be running its termination grace period. It may still be mid-write to Redis or Mongo. It may hold a lease that has not expired. And a replacement Pod can reuse the same name — deterministic naming, which solves the convergence problem in section 5, creates exactly this problem here.

So the kill flow does not return when the delete is accepted. It returns when the exact old Pod UID has disappeared:

acquire RuntimeInstanceId lifecycle gate
    -> drop the convergence cache entry
    -> stop any direct port-forward
    -> load the exact cached Pod specification
    -> delete Service and Pod
    -> wait for exact Pod UID disappearance      <-- the actual boundary
    -> delete the runtime Gateway route (best effort)
    -> remove capacity
    -> unregister the runtime instance

Name-based checking would pass the moment a replacement Pod appeared with the same name, which is the precise moment the guarantee is least true. UID-based checking is the only version of this that means anything.

The test boundary asserts the same property from the outside: deletion was requested; the old UID is gone; the old runtime writes no further progress after KillAsync returns; unsafe capacity has been suppressed; and only then does recovery begin.

Without that strict boundary you get the failure mode that quietly destroys determinism: the old runtime and its replacement both mutating the same execution. Not a crash. Not an error. No stack trace, no alert, nothing red anywhere. Two writers on one durable ExecutionId, producing a history that cannot be replayed — because it never happened in one order.

That is the failure I care most about, and it is the reason the kill path is more paranoid than it looks. A system that loses work tells you it lost work. A system that has two writers tells you nothing, and the corruption surfaces months later as a replay that does not reproduce.

8. Kubernetes does not perform execution recovery

This deserves to be stated flatly, because “Kubernetes restarts things” is the single most common wrong assumption about this milestone — and it is wrong in a way that sounds sophisticated.

The process-host version of this recovery proof is described in *When the Process Dies, the Execution Doesn’t: Proving Multi-Tenant AI Runtime Crash Recovery*. The Kubernetes work keeps the same durable recovery model and the same reconciler ownership. It replaces the host lifecycle and strengthens the failure boundary without moving execution recovery into Kubernetes-specific infrastructure code.

Kubernetes can replace infrastructure. It has no idea what a SharedRunId is. It does not know about LocalRunId, ExecutionId, DAG progress, replay metadata, tenant isolation, recovery forensics, or whether a given piece of work must resume or restart. A restarted Pod does not resume the old in-memory execution; it starts a fresh runtime instance with no memory of the previous one. Everything durable lives in control-plane-owned stores — and the control plane owns every decision about what to do with that state.

Recovery ownership stays exactly where it was for process hosts:

RuntimeInstanceHealthReconciler
    detects the unsafe runtime instance
    suppresses its capacity so nothing new is routed there

AiRuntimeExecutionRecoveryReconciler
    enumerates work assigned to the failed runtime
    classifies each item and chooses a recovery mode

Runtime Host Manager / provider scale-out
    creates or selects replacement capacity

HTTP or gRPC provider
    issues runtime commands against the replacement runtime

The two recovery modes are genuinely different operations and must not be described as one:

InFlightExecution — the DAG had started. Recovery resumes the same durable ExecutionId. No second logical execution is created for the same in-flight DAG. This is the mode that makes replay meaningful: the execution has one identity and one history, regardless of how many hosts it outlived.

LocalQueued — the work was assigned to the failed runtime but never started. The dead runtime’s local queue is volatile and is never treated as durable recovery truth. Recovery redispatches through the durable SharedRunId, emitting SharedRunRequeuedForLocalQueuedRecovery, and the work lands on a replacement runtime with a new LocalRunId. This is a redispatch, not a same-ExecutionId resume.

Collapsing those two into “it recovers” is how you ship a system that silently duplicates work — because the resume path and the redispatch path have opposite correctness conditions. Resume must not create a second execution. Redispatch must create a second local run. Get them backwards and you either lose work or do it twice, and both are invisible until someone reconciles a ledger.

Forensic evidence stays correlated by ForensicsId and RuntimeFailureIncidentId, linking the failed runtime instance, the failed local run, the shared run, the execution, the replacement runtime, the recovery classification, and the timeline. That correlation is the artefact that lets you answer, months later, what happened to this specific piece of work — which is a question that regulated environments ask and "Kubernetes restarted the Pod" does not answer.

9. Failure discovered under parallelism: the HTTP Kubernetes endpoint race

Everything above describes an architecture that was, I thought, already correct. gRPC Kubernetes had passed. HTTP Kubernetes had passed at parallelism 1 and 2. Then HTTP Kubernetes ran at parallelism 5, and one run out of forty-five sat at QueuedGlobally with FailureReason=http-endpoint-missing.

One out of forty-five. That ratio is the whole reason this section exists, and it is why I want to describe how it was found and not only what it was — because the diagnostic path is more reusable than the bug.

What the structured failure bought me. http-endpoint-missing is not a generic error. It is a specific dispatch-time classification: admission selected capacity for the runtime, but the HTTP provider could not resolve a usable command endpoint from the selected descriptor. That single string eliminated most of the search space before I opened a log. It was not a Pod problem, not a scheduling problem, not a route problem — the control plane had a capacity record and the record was wrong. The taxonomy of structured failure reasons — configuration missing, Pod specification failure, host creation failure, readiness timeout, endpoint resolution failure, port-forward startup failure, runtime command readiness failure, Gateway readiness or route failure, publication failure — is the thing that turns "it hangs sometimes" into a bisected problem.

Why parallelism surfaced it. The race requires two independent publishers to interleave inside a narrow window. Lower-parallelism runs did not expose that interleaving consistently. At parallelism 5, with fifteen tenants building inventories simultaneously on a single-node cluster, startup, heartbeat, and publication operations overlapped often enough to make the latent race observable. The concurrency did not create the bug. It created the sample size.

The cause was two publishers writing the same capacity descriptor.

Publisher A — the Kubernetes host publisher. Runs once, after readiness. Knows the externally routable Gateway or port-forward endpoint. Publishes transport.endpoint, the runtime command endpoint, tenant ownership, and Kubernetes route metadata. This is the rich descriptor.

Publisher B — the runtime Pod heartbeat, via AiRuntimeInstanceRegistrationHostedService. Runs periodically. Knows the runtime's own view of itself: worker count, queue depth, available slots. Does not know the external endpoint, because the Pod has no way to learn what address the control plane reaches it on.

RedisAiRuntimeInstanceCapacityStore originally replaced the whole descriptor on write. It did not merge metadata. Under load, the interleaving is obvious in hindsight:

t0   heartbeat reads runtime state          [no external endpoint known]
t1   Kubernetes publisher writes rich descriptor   [endpoint present]
t2   the already-in-flight heartbeat writes its own descriptor
                                                   [endpoint GONE]
t3   admission sees capacity with no endpoint
t4   dispatch fails: http-endpoint-missing

The gRPC Kubernetes path was already protected by an earlier guard: a gRPC runtime without a usable endpoint could not advertise usable capacity, so the stale write produced unusable-but-honest capacity rather than usable-but-broken capacity. HTTP had no equivalent guard. The underlying metadata regression was possible in both paths, but only HTTP could turn it into an unsafe dispatch decision.

That distinction matters. gRPC was safe against this dispatch failure, but the comparison revealed that its guard was masking a broader publication invariant that needed to hold for both transports.

The fix touched exactly three production files, and it is three separate ideas.

One: no endpoint, no capacity. Both HTTP and gRPC Kubernetes runtimes now require a usable transport endpoint before capacity can be admitted. Without one:

CanAcceptRun      = false
AvailableRunSlots = 0

The runtime is still registered and still visible. It is simply not schedulable. That is the honest state, and it lets diagnostics say why rather than making the runtime disappear.

Two: publication is monotonic. The Redis capacity store keeps its fast path for normal descriptors, but uses an optimistic compare-exchange path specifically for Kubernetes HTTP/gRPC descriptors that arrive without an endpoint. A stale, endpoint-less descriptor can no longer replace a richer one that already carries the transport endpoint, endpoint aliases, route metadata, and tenant ownership. Note the narrowness: the slow path applies only to the descriptors that can cause the regression. Everything else keeps the fast write.

Three: HTTP gets the proof gRPC already had. The Gateway readiness check for HTTP now sends a real runtime command — GetQueueStatus through /runtime-instance/commands — with the configured runtime routing header, and validates that the responding runtime is the expected RuntimeInstanceId. Not "did I get a 200." Did I reach the runtime I meant to reach.

The generalisable lesson is the second point, and it is not a Kubernetes lesson at all:

A later write must never make a runtime less routable than an earlier one.

Any component that publishes partial state must merge into richer state, not replace it — because the component with the narrowest view is usually the one that writes most frequently.

Heartbeats are the archetype. They are periodic, they are cheap, they are written by the component closest to the resource, and they know the least about how that resource is reached. Every system I have worked on that had a heartbeat and an out-of-band enrichment step eventually had this bug. The fix is always the same shape: make the frequent writer a merge, not a replace.

10. What stayed the same / what changed

Stayed the same Changed Admission and tenant-visible capacity rules How a runtime host is created and destroyed Durable scale-out request, store, and watcher Number of readiness layers before dispatch IAiRuntimeHostManager contract Endpoint resolution and route lifecycle Identity model: SharedRunId / LocalRunId / ExecutionId / RuntimeInstanceId Crash boundary semantics — UID disappearance, not process exit Capacity descriptor model Who owns externally routable endpoint and route publication Crash recovery, replay, ledger, trace, forensics Cleanup ownership and failure-path resource deletion Tenant isolation enforcement Idempotency requirements on host creation DAG execution semantics Recovery timing — slower, more staged, more observable

HTTP and gRPC parity follows the same shape. Identical: admission, scale-out request, Host Manager, Kubernetes resources, runtime identity, capacity model, crash recovery, replay, ledger, trace, tenant isolation. Different: the command client, HTTPRoute versus GRPCRoute, the routing probe, and protocol-specific endpoint validation.

HTTP Kubernetes is not a second runtime architecture. It is transport parity over one host lifecycle. That is a claim the test matrix has to earn, and section 12 is where it earns it.

11. Fake client versus Kubernetes SDK

Both client modes exist because they prove different things, and conflating them produces confident, worthless tests.

FakeAiKubernetesRuntimeHostClient is an in-memory lifecycle simulator. It proves Host Manager strategy selection, metadata and request composition, convergence behaviour, structured failure classification, and ownership-aware cleanup. It is fast, deterministic, and needs neither a cluster nor Redis. It creates no Pod, no Service, no routable NodePort — and, after a bug fix in this milestone, it emphatically does not start a real kubectl port-forward, because a forward to a Service that does not exist can never become reachable, and a test that waits for it can only ever time out.

That bug is worth naming because of what it represents: a fake that was faithful enough to trigger real side effects. A test double that reaches outside the test is not a double. It is a slower, less reliable version of the real thing.

A Fake-client test is therefore not transport evidence. It cannot be, by construction.

KubernetesSdkAiKubernetesRuntimeHostClient creates real Pods and Services, polls real readiness, validates identity when resources already exist, issues real deletes, and confirms exact UID disappearance. It is slow and it needs a cluster. It is the only thing that can prove routing, real workload execution, and real crash recovery.

The rule I now apply to the whole suite: every test must be able to state which boundary it proves, and no test may be cited as evidence for a boundary it does not touch. Fake lifecycle tests, SDK resource tests, transport-readiness tests, Gateway tests, and production crash-recovery tests are not interchangeable evidence, and a green suite that mixes them without saying so is a suite that will let a section-9 bug through.

12. The proof scenarios, and what they actually prove

The test progression escalates deliberately, and each step adds exactly one new failure mode:

fake Host Manager scale-out        strategy selection, metadata composition
Kubernetes SDK scale-out           real resources
runtime readiness                  application layer
routable endpoint                  transport layer
real work dispatch                 the runtime actually executes
single Pod crash                   the failure boundary
two impacted tenants               concurrent recovery
impacted + safe tenant             recovery containment
controlled parallelism             interleaving
stability loops                    non-flakiness

The workload is not a smoke test. Each run executes a 50-step DAG with a deliberately flaky step at 500 ms intervals, and the kill lands mid-execution, after 25 completed steps — so every impacted runtime dies with real in-flight DAG progress plus queued work behind it, against MongoRedis persistence and DurableMongo observability.

The final HTTP Kubernetes proof, after the endpoint-race fix:

parallelism                 5
simultaneous scenarios      5
tenants                     15
submitted runs              45
impacted tenants            10
safe tenants                5
per scenario                2 impacted tenants + 1 safe tenant
per runtime inventory       1 in-flight execution + 2 local queued runs
in-flight recovery          resumed on the same ExecutionId
local queued recovery       redispatched via SharedRunId
safe tenants                absent from all recovery evidence
duration                    ~8.6 minutes

The safe tenants are the load-bearing assertion. A recovery system that recovers too much is as broken as one that recovers too little — and it is far harder to detect, because over-recovery looks like success from every angle except the ledger. The only way to catch cross-tenant contamination is to run uninvolved tenants through the same crash window and assert that their forensics, ledger, and trace stay empty of recovery events. Five safe tenants, five crash windows, zero recovery records. That assertion is worth more than the forty-five successful runs.

What this proves, and what it does not prove

Proves: that a Kubernetes-hosted runtime executes real DAG work; that Pod deletion establishes a strict, UID-exact failure boundary; that in-flight executions resume on the same ExecutionId across a host boundary; that local-queued work is redispatched through SharedRunId without duplication; that tenant isolation holds through a crash; and that HTTP and gRPC reach the same recovery outcomes over different transports. The successful HTTP Kubernetes parallelism-5 run validates the corrected path; repeated stability loops remain the stronger non-flakiness proof.

Does not prove: anything about production cluster behaviour. This ran on Minikube and Docker Desktop, single-node, with the control plane outside the cluster reaching runtimes through a port-forwarded shared Gateway. Local single-node performance is not production cluster performance and the ~8.6-minute figure should not be read as a benchmark of anything. Parallelism above 5 is not validated. Cluster-level hardening — real ingress, network policy, image distribution and pull latency, node pressure, eviction behaviour, multi-node scheduling, PodDisruptionBudgets, resource quotas — remains operational work I have not done.

I am not claiming this has been run on a large production Kubernetes cluster, because it has not been. What I am claiming is narrower and, I think, more useful: the host contract is now explicit enough that production-cluster validation can focus primarily on operational and scale behaviour — while remaining open to architectural gaps that only a multi-node environment may expose.

13. Operational lessons

These are the ones I would give someone building a runtime that hosts durable work on Kubernetes, in the order I wish I had learned them.

On readiness

• Pod Running is not runtime readiness, and no single Kubernetes signal ever will be. Readiness is a conjunction, and each conjunct needs a named owner.
• Prove reachability with a real command that returns the callee’s identity — not a TCP connect, not a 200, not a health endpoint. “Did I reach the runtime I meant to reach” is the only question that matters.
• Capacity must never be published before routability is proven.

On publication

• Publication must be monotonic. A later write must never make a runtime less routable than an earlier one.
• The component that writes most often usually knows least. Design its writes as merges, not replacements.
• A component should only publish facts it can actually know. A Pod cannot know its external endpoint, so it must not be the thing that publishes it.

On failure

• Resource deletion is not automatically a deterministic crash boundary. Bind the boundary to exact resource identity (UID), not to the delete call returning.
• Deterministic naming and crash boundaries pull in opposite directions. Names are for convergence; UIDs are for boundaries. Do not use one for the other’s job.
• Structured failure reasons are a debugging multiplier. http-endpoint-missing eliminated most of a search space before I opened a log.

On boundaries

• Host lifecycle and execution recovery must stay separate, or recovery logic slowly migrates into infrastructure configuration where it cannot be replayed, tested, or audited.
• Provider identity and host identity must remain explicit and separately queryable. provider.name answers how do I send commands; host.provider answers where does this thing live. Conflate them and you cannot express "same protocol, different host," which is the entire point.
• Infrastructure metadata is evidence, not authorization. A namespace is not a tenant boundary.

On testing

• Every test must state which boundary it proves. Fake lifecycle tests, SDK resource tests, transport tests, and recovery tests are not interchangeable evidence.
• A test double that starts real external processes is not a double.
• Tests must prove the failed runtime stopped writing before replacement execution continues.
• Safe tenants belong in the proof, not in an optional extra assertion. Over-recovery is the harder failure to see.
• Concurrency does not create races. It creates sample size. If a race is theoretically possible, raise parallelism until it is observable — and assume the paths that pass are lucky until you can name the guard that protects them.

14. What comes next

Kubernetes hosting closes one major boundary, but it also exposes the next scale questions more clearly. The immediate roadmap is not another transport or another host type. It is the work required to make the same deterministic protocol efficient under much larger and more sustained concurrency.

High-concurrency hardening. The current scenarios prove correctness under controlled parallel failure, including tenant isolation and safe-tenant non-impact. The next proof must push admission, dispatch, recovery, ledger publication, and concurrency enforcement much harder. The goal is not merely to increase a test parameter — it is to verify that provider, model, operation, execution, tenant, and control-plane scopes remain genuinely independent under pressure; that backpressure is explicit rather than emergent; and that no metadata key silently becomes a global bottleneck. Section 9 is the argument for this milestone: the bug that mattered was invisible at low concurrency and obvious at parallelism 5, and there is no reason to believe that is the last one.

Runtime pool management. Today the Host Manager can create and replace process or Kubernetes runtime instances, but every scale-out request is a reason to create a fresh host. A pool manager should know whether a runtime is idle, leased, tenant-affine, shared, dedicated, draining, unhealthy, or eligible for reuse. Reusing warm process and Pod capacity should cut startup latency and cluster churn substantially — but only if reuse cannot weaken tenant visibility or blur the deterministic recovery boundary, which is precisely the hard part. A pooled runtime that serves two tenants in sequence must leave no trace of the first.

Redis cluster catalogue and partition-aware routing. The current Redis stores already carry tenant and execution context correctly, but operating at very large scale requires an explicit catalogue of where durable keys live. That catalogue must route tenants, runs, execution records, queues, leases, and indexes to the correct Redis cluster or partition while preserving atomic operations and avoiding accidental cross-slot assumptions. The long-term target is not “one Redis instance with more memory.” It is a partition-aware control plane that can locate and operate on millions of tenant and execution records without losing deterministic ownership.

These are deliberately separate milestones. Concurrency hardening proves the protocol under pressure. Runtime pooling improves host efficiency. The Redis catalogue makes state placement explicit. Combining all three into one refactor would hide which invariant failed — and this article is largely a demonstration of how much easier the work becomes when you can name the invariant that broke.

15. Closing argument

Moving from child processes to Kubernetes Pods did not require rewriting the deterministic execution protocol. The DAG engine, the identity model, the replay path, the ledger, and the forensics kept the same durable semantics. The recovery model did not have to migrate into Kubernetes-specific code.

What it required was making the host contract explicit: separating lifecycle from readiness, readiness from routing, routing from identity, identity from capacity publication, deletion from the crash boundary, and the crash boundary from recovery ownership.

Every one of those separations was already true in the process host. They were just invisible, because in a single process they all become true in the same instant. Kubernetes did not introduce them. It introduced latency between them — and latency between two facts is the only thing that ever forces you to admit they were two facts.

That is the honest summary of this milestone. I did not add distributed-systems complexity by adopting Kubernetes. I discovered the complexity that had always been there, hidden behind the coincidence that a child process is its own endpoint.

Once the separations were named, process-hosted and Kubernetes-hosted runtimes could run the same durable protocol, recover through the same reconcilers, and produce the same evidence. The host became a replaceable lifecycle strategy rather than part of the execution semantics. That was the goal.

Kubernetes did not make the runtime deterministic. Determinism made Kubernetes replaceable as a hosting strategy.

16. Raw execution evidence

The summaries above are extracted from the actual integration-test output, but the complete logs are available for readers who want to inspect the proof beyond the selected excerpts.

The raw output includes the full runtime lifecycle: tenant-scoped scale-out, Pod creation and readiness, real DAG progress before termination, exact failed-runtime identification, replacement-capacity selection, same-ExecutionId recovery for in-flight work, SharedRunId redispatch for local-queued work, replay validation, recovery forensics, tenant-scoped ledger queries, and the final safe-tenant non-impact assertions.

The HTTP Kubernetes proof records successful recovery for all expected impacted work, validates the complete control-plane causal chain, and reports zero foreign-tenant or safe-tenant recovery leakage.

Download the complete HTTP Kubernetes crash-recovery log

The gRPC Kubernetes proof exercises the same durable recovery contract through the gRPC transport: both impacted tenants recover their assigned work, the safe tenant remains outside the recovery path, and the tenant-scoped ledger and control-plane causal-chain validations complete successfully.

Download the complete gRPC Kubernetes crash-recovery log

These files are intentionally provided as raw test output rather than edited demonstrations. They contain the timings, runtime identities, execution identities, recovery classifications, forensic timelines, ledger counts, and safety assertions produced by the running system.

References

• Repository: github.com/mmarano2k14/deterministic-ai-runtime
• Canonical architecture doc: [docs/ai/kubernetes-runtime-host-provider.md](https://github.com/mmarano2k14/deterministic-ai-runtime/blob/feature/kubernetes-runtime-host-provider/docs/ai/kubernetes-runtime-host-provider.md)
• Related: [runtime-instance-provider-model.md](https://github.com/mmarano2k14/deterministic-ai-runtime/blob/feature/kubernetes-runtime-host-provider/docs/ai/runtime-instance-provider-model.md) · [runtime-discovery-registry-capacity.md](https://github.com/mmarano2k14/deterministic-ai-runtime/blob/feature/kubernetes-runtime-host-provider/docs/ai/runtime-discovery-registry-capacity.md) · [testing-strategy.md](https://github.com/mmarano2k14/deterministic-ai-runtime/blob/feature/kubernetes-runtime-host-provider/docs/ai/testing-strategy.md)
• Prior crash-recovery article: *When the Process Dies, the Execution Doesn’t: Proving Multi-Tenant AI Runtime Crash Recovery*
• Prior distributed-identity article: *Why One RunId Is Not Enough: TenantId, SharedRunId, LocalRunId, and ExecutionId in a Distributed AI Runtime*