Wire the LLM Plumbing Once. Every Agent Session Inherits It.

Image created by the author using Midjourney

Every agent session building software that calls an LLM starts by re-laying the same plumbing before any real work can begin: config loading, model wiring, vector store setup. Identical every time, rebuilt as if never solved. Every token spent on infrastructure that already exists is a token the actual work never gets. This is what that plumbing is, and how to wire it once so every session inherits it instead.

While building a RAG pipeline with three distinct stages, each stage created in a fresh context session. Every session rebuilt the same plumbing before the actual work could start. The first ingested documents into a vector store. The second queried it. The third built the retrieval layer that answered questions from the ingested data. The problem being solved was different each time. The pre-conditions were identical.

The traditional argument for a shared layer is code reuse: the same logic behind canonical transformation layers in data engineering. One implementation. Consumers inherit. Override only what differs. That still applies. But it understates what’s actually at stake now.

When a coding agent can produce a tailored, working implementation from scratch in minutes, re-implementation stops being expensive. The cost of duplication shrinks.

What doesn’t get thinner is context. A session has a fixed window.

In a typical 50-turn agentic session, context grows from roughly 5,000 input tokens at turn one to 25,000–35,000 by turn 30. That window has to carry everything the session has read, written, and encountered so far.

The plumbing is identical every time: same config shape, same LLM provider wiring, re-derived as if never solved. Every session rebuilds what the last one already built, and those tokens never reach the actual problem.

The shared primitives layer is an answer to that. Not primarily a code quality decision. A token economy decision.

What kept getting in the way

The ingest stage loaded documents, chunked them, embedded them, and persisted the results to a vector store. Before any of that could run, there were setup problems to solve first: how to load and validate config, which model to call and how to wire it, how to set up the vector store client and storage context, what the default retrieval knobs should be.

Those aren’t document-loading concerns. They’re pre-conditions. Infrastructure that has to exist before the actual work can begin.

The next two stages answered questions from that knowledge base: hybrid search, synthesis, citations. A different problem entirely. Same pre-conditions, resolved again from scratch, slightly differently, because each development session starts clean with no memory of what a previous session decided for a different stage.

It didn’t matter what came after. Classification, document generation, chart synthesis, multi-turn exploration: each started from the problem at hand and hit the same blockers before any of it could run. And one more appeared consistently: every workload needed a command line, and each was building its own argument contract from scratch.

That was the moment the primitives became visible, not as a design choice, but as a recognition that the same problems kept getting in the way. The question became: what if those things were solved once?

The primitives

The primitives share a common configuration schema every workload inherits. It captures the execution options every workload needs to decide before it can run: which model, which provider, where knowledge is stored, how retrieval behaves, how to chunk.

Each workload specifies only what it changes, a shared baseline handles the rest. The result is a complete, deterministic runtime view with no decisions left implicit.

• Shared baseline: one file per workload, covering model, provider, storage, retrieval, and chunking with known-good defaults
• Module override: typically 4–8 keys, only what this workload changes
• Merged result: a complete, deterministic runtime view, ready to drive the primitives

Built on that config layer, the primitives provide the entry points a workload needs before any actual work can start:

• Config loading: resolves the merged runtime view from baseline and module override.
• Model wiring: one call to produce the embedding model and one to produce the LLM object, both driven by config rather than provider SDK code in the calling code.
• Portable LLM calls: one caller that routes to local or cloud providers and normalises tuning keys.
• Framework-native LLM wrapper: the same provider routing exposed through a framework-friendly interface.
• Vector store setup: a small set of helpers to create the client, attach it to a storage context, and list or discover collections consistently.
• Query engine: one call that combines retrieval and synthesis, driven by the same query knobs.
• CLI contract: parse a YAML spec so the command’s --help stays authoritative.

Provider-portable LLM calls and model wiring

A single portable interface handles all providers. The provider is an environment variable: local by default (Ollama runs natively on Apple Silicon, within the 18GB unified memory constraint), any cloud provider for production. The calling code is identical either way. It normalises tuning options across providers so the caller never has to know which provider is handling the request or how it names its parameters.

The example below runs identically against a local Ollama server or a cloud API; the only difference is what AI_AGENT_CORE_PROVIDER is set to:

[embed]Provider-portable LLM call, identical invocation whether the endpoint is local Ollama or a cloud API.

For workloads that need a framework-native LLM object rather than a direct call (LlamaIndex, for instance), a companion wrapper exposes the same provider logic through the interface those frameworks expect: same environment variable, same provider routing, no additional configuration. Model wiring follows from config, one call per model type produces the model objects the framework expects. The config drives it; the caller receives ready-to-use objects.

Vector store and knowledge base setup

For solutions that need a knowledge base, the primitive provides a small set of helpers for vector store setup: create the client, create the storage context, and list collections. The primitive resolves persist path, storage context, and collection naming consistently from the shared config. Workloads that make no retrieval calls don’t need it.

Retrieval defaults and query engine

For workloads that query a knowledge base, the config schema carries opinionated starting points: hybrid search on, top-k at five, reranking off, context budget capped. They emerged from building each one from scratch, noticing what kept working, then distilling the pattern into a shared baseline so subsequent workloads inherit rather than re-derive. Hybrid search, combining BM25 sparse retrieval with dense vector search, is the baseline because it consistently improves recall for exact terms and citations without application-level tuning. On this platform, 7–9B models are the working size. That memory limit makes larger models impractical. Smaller models miss exact-term matches that a 70B model would catch: rule references, regulation codes, entity names. Hybrid retrieval compensates structurally rather than by upgrading model size. That held across every workload I built on this platform; highly specific or multilingual corpora may need tuning before treating it as a safe default.

Reranking is off by default because it adds per-query compute; it belongs in workloads where precision matters more than throughput. I left it off for most of the platform and turned it on only where retrieval quality became the bottleneck. That ordering matters. Each workload overrides these when it has a reason. The baseline is correct in the absence of one.

The primitive extends to a query engine that combines the retriever and LLM synthesis into a single call, the full retrieval loop driven by the same config knobs behind one entry point. Workloads that need the full path use it; those that only need the retriever use the retrieval primitive directly.

Config-driven CLI contract

Runnables share a single CLI argument contract, defined in a YAML spec. The entry point parses it with a single call. The --help output is authoritative, there is no separate CLI documentation to maintain or drift from the actual interface. For the first two commands, a shared YAML spec looks unnecessary. By the fifth, changing a CLI contract is a ten-minute edit and a working --help.

Why documentation is not the constraint

Once these primitives existed, the problem was not whether they were documented. The problem was whether a new development session, building the next implementation, would use them or rebuild them.

Agent-driven development is fast. It is also stateless. A session building a classifier starts from what it needs to do and works outward. It has no memory of what the previous session resolved for a different workload. It doesn’t know a shared config loader exists. So it builds one. Slightly differently. Shared infrastructure falls between session scopes; no single session is responsible for it, so none maintain it.

Documentation doesn’t solve this. A rule that says “use the shared layer for config loading” competes with all the domain-specific context in the session. Under enough session length and task-specific signal, soft guidance loses.

The structural fix is an import dependency. When a new implementation must import the shared layer rather than build its own, the primitive is used regardless of what the session remembers, or doesn’t. This is not a process problem or a discipline problem. Statefulness is the architecture of how coding sessions work; no amount of better documentation or guidance changes that.

The same logic as the canonical layer in data engineering.

The primitives held when it expanded

I added classification and visualisation modules after the primitives layer was in place. Both are operationally distinct from the earlier ones. Classification added taxonomy constraints, confidence gating, and structured output; visualisation added chart insight generation and formatted response schemas. Different problems, different output shapes.

Both hit the same pre-conditions before any of the actual work could begin.

Neither required a new primitive.

That is the test: not whether the primitives work for the workloads they were discovered through (they will), but whether they hold when the next workload is different enough to matter. These two were different enough.

Before any session builds: check what already exists

The package ships a primitives index, organised by need, not by what the library is called. A new session building the next implementation starts by running ai-agent-primitives-index. Each entry maps a common need to the exact import, annotated with what the primitive provides in one line. Before writing any config loading, model wiring, or vector store setup, the session checks here first. If what it needs is listed, it uses it. If not, it belongs in the workload.

[embed]Primitives index — ten primitives, each mapped to the need it covers.

The primitives ship as ai-agent-core. Local setup is Ollama for model serving plus uv add ai-agent-core; ChromaDB and LlamaIndex come as dependencies. Switching to a cloud provider is an environment variable change, not a code change.

The platform the primitives enable

• Ingest — PDF/XML/Excel → ChromaDB + BM25 + JSON sidecars
• Query — RAG query runner with grounded answers and source citations
• Retrieve — Quality-gated enrichment; bounded agentic fallback loop
• Classify — Taxonomy-constrained classification; confidence gating; resumable runs
• Visualise — LLM-driven chart insight generation; Plotly and Vega-Lite rendering
• Explore — Multi-turn LangGraph exploration agent
• Orchestration — UI + tool-calling agent + MCP server over the same orchestrator
• Substrate — Deterministic knowledge graph from JSON sidecars (no LLM)

Eight implementations. The substrate module makes no LLM calls and carries no primitives dependency; it builds on a graph library directly. Every other implementation uses the same primitives as its foundation.

What I learned building this

The primitives became visible through friction, not design.

The compounding started at the first stage, even though it only became visible at the third. By the time the pattern was clear, the cost had already been paid several times over. It should have existed before the first stage, not after the third made the pattern visible.

Declaring it as a package dependency in each workload was what made it structural. A new session building the next implementation got the primitives whether it went looking for them or not. The agent didn’t change. What changed was how much of the problem it arrived at already solved.

AI Disclosure: The author originated, researched, and drafted this content from direct implementation experience. AI was used solely to refine prose and formatting.

This story is published on Generative AI. Connect with us on LinkedIn and follow Zeniteq to stay in the loop with the latest AI stories.

Subscribe to our newsletter and YouTube channel to stay updated with the latest news and updates on generative AI. Let’s shape the future of AI together!