🦾 Building a Modular AI Micro-Agent Architecture on Kubernetes: From Ollama to KServe (and Beyond)

How to start simple, scale smart, and keep your MLOps future-proof

🚀 Introduction

In 2025, AI is everywhere — and so are LLMs and AI agents. But building a scalable, maintainable, and production-grade AI backend (without drowning in complexity from day one) is still a challenge.

Here’s my favorite strategy: a step-by-step, modular approach for designing an AI micro-agent architecture, deploying everything on Kubernetes, and preparing for future upgrades — so you can move fast now, and scale or pivot later!

🧩 Hosted Inference services (model serving)

This article specifically focuses on how to design and deploy modular, scalable, and observable AI inference services — hosted directly on Kubernetes for your custom models.

It’s especially relevant for teams who want (or need) to self-host their AI models (ML/LLM)— either for privacy, cost, regulatory, or flexibility reasons — and can’t (or don’t want to) rely on SaaS APIs like OpenAI, AWS Bedrock, or Azure OpenAI.

🏗️ Why Micro-Agents?

A “micro-agent” architecture means:

• Each agent/service is small, focused, and independent.
• Agents interact via HTTP APIs (hello, cloud native).
• You can scale, update, or replace any agent with zero impact on the rest.
• You’re never locked in — add more tools or models as your project grows.

Ten years ago, most backend systems were monolithic services — large, tightly coupled apps that did everything. Then came the rise of microservices: small, independent, and specialized services working together. Today, we see the same trend emerging with AI agents:

• Early “monolithic” agents tried to bundle all capabilities into a single huge bot.
• Now, we’re moving to micro-agent architectures: many focused agents, each handling one skill or workflow, collaborating via APIs — just like microservices.

This shift introduces new challenges, just as it did for microservices:

• The need for unified API gateways (AI Gateway, LLM Gateway, Envoy, etc.)
• Observability, tracing, and monitoring across all agent interactions.
• Smart routing, versioning, and scalable deployments.

That’s why this article focuses on how to design a modular, scalable, and observable AI backend with micro-agents, Kubernetes, and modern model-serving tools — plus strategies to tackle the new architectural challenges ahead.

🟢 Step 1: Ollama on Kubernetes (Simple & Fast)

Ollama is your self-hosted LLM server (supports Llama, Mistral, etc.), exposing an OpenAI-compatible API.

How does it look?

+-------------------+       +---------------------+
|      Agent        | <---> |   Ollama LLM Server |
+-------------------+       +---------------------+
        |                         |
   (Pod in k8s)              (Pod in k8s, Helm)

• Each micro-agent makes HTTP calls to Ollama.
• Ollama serves your LLMs (you choose which ones to install).
• Deploy both easily via Helm chart or manifests.

Why start this way?

• Dead simple to iterate and debug.
• Minimal stack, fast setup.
• Just update Ollama’s models, and all agents benefit instantly.

🚀 Alternative: Serverless Scalability with Knative/KEDA + Ollama

If you want to scale your Ollama LLM backend “serverlessly” — without bringing all of KServe — you can deploy Ollama as a Knative Service or with KEDA.

• Knative Serving provides scale-to-zero, instant autoscaling, and advanced routing for any containerized app (including LLM servers).
• You get cloud-native scaling for Ollama, just like with KServe’s “serverless” mode, but lighter for simple use cases.

+-------------------+
|   User/Agent      |
+-------------------+
        |
        v
+-------------------+ (scales-to-zero with knative, simple scaling with keda)
|   Knative/KEDA    |---------------------> Ollama LLM Server
|   Ollama Service  |
+-------------------+

Best for:

• When you want instant scaling for LLMs
• When you don’t need all the extra ML features of KServe

🟡 Step 2: KServe (Scale & Flexibility)

KServe is more than just a serving platform for LLMs. It’s a universal model serving solution for all ML workloads:

• LLMs (Llama, Mistral…)
• Vision models
• Classic ML (scikit-learn, XGBoost…)
• Any framework: PyTorch, TensorFlow, ONNX, HuggingFace, Triton, and custom containers.

Scalabitly

You have two main options for scalable, production-grade serving with KServe:

• KServe in serverless mode (with Knative): Get instant autoscaling and scale-to-zero — just like a cloud function for your models.
• KServe in classic Deployment mode (with KEDA): Use Kubernetes-native deployments, gain fine-grained control, and leverage event-driven autoscaling — without the Knative stack.

This flexibility lets you tailor your infrastructure — whether you want cloud-style, cost-efficient serverless serving, or always-on, “classic” Kubernetes pods with predictable behavior.

How does it look?

+-------------------+
|       Agent       |
+-------------------+
         |
         v
+----------------------+
|      KServe/KEDA     |
| (LLM/ML Inference)   |
+----------------------+
         |
   [Multiple Models: Llama, Mistral, Custom...]

• Replace (or complement) Ollama with KServe for model serving.
• KServe exposes HTTP endpoints for each model, LLM or not.
• Agents can switch between models dynamically.
• Get auto-scaling, canary releases, monitoring, and more “for free.”

Why move to KServe?

• Not LLM-only: KServe is designed for all machine learning models — text, images, tabular, anything!
• Universal serving: Deploy multi-modal, multi-framework models in one system.
• Seamless multi-model deployment (LLMs, embeddings, classifiers, vision…)
• Scale-to-zero, easy versioning, rollbacks, advanced monitoring
• Production-grade robustness

🟠 Step 3: Bring in Kubeflow for Full MLOps

Need pipelines, model registry, automated retraining, or advanced team workflows? Time to add Kubeflow.

How does it look?

+-------------------+       +-----------+
|      Agent        | <---->| Kubeflow  |
+-------------------+       +-----------+
         |                         |
         v                         v
   [KServe / Inference]      [Pipelines, Registry, Notebooks...]

• You keep your KServe serving stack.
• Add Kubeflow components as needed (pipelines, notebook, registry, etc).
• Go from fast prototyping to true MLOps without a migration headache.

🧠 Why This Evolution Makes Sense

• Start fast, stay lean: No “MLOps tax” until you need it.
• Micro-agents = full modularity: Each part can evolve independently.
• Kubernetes-native = cloud portable: Works on any cluster — cloud, on-prem, hybrid.
• No lock-in: Changing LLM backends or adding new models = config change, not a rewrite.

🚦 Pro Tips for a Smooth Transition

• Never hardcode endpoints: Use config files or env vars for LLM/KServe/Ollama endpoints.
• Keep agents stateless: Easier to scale, swap, and debug.
• Add monitoring/logging early: Prometheus, OpenTelemetry, whatever fits your stack.
• Document APIs: For future you, your team, and anyone else plugging into your micro-agents.

🧩 Bonus: Adding an Envoy AI Gateway for Unified LLM & Model Access

As your architecture grows (more agents, more LLMs, more ML models), routing, securing, and managing all those endpoints becomes a challenge. Enter Envoy AI Gateway (and similar LLM/API gateways):

Why use an AI Gateway?

• Unified API layer: Aggregate access to Ollama, KServe, HuggingFace, vLLM, Triton, or any other model server behind a single endpoint.
• Protocol adaptation: Expose your LLMs and ML models via a uniform OpenAI/ChatGPT-compatible API, no matter what’s running underneath.
• Security & auth: Centralize authentication, rate limiting, and logging for all your AI traffic.
• Advanced routing: Implement load balancing, A/B testing, multi-model routing, fallback logic, and more.
• Observability: Built-in support for metrics, tracing, and monitoring across your entire AI serving fleet.

How does it fit in your stack ?

+-------------------+
|       Agents      |
+-------------------+
        |
        v
+-------------------+
|  Envoy AI Gateway |  <-- (single endpoint: /v1/chat/completions)
+-------------------+
   |      |     |
   v      v     v
 [Ollama][KServe][Other LLMs/Models]

🏁 Conclusion: AI Infra That Grows With You

Start with Ollama for speed and simplicity. Upgrade to KServe when you need scale, flexibility, and more models. Adopt Kubeflow for full-on MLOps and team workflows.

Don’t over-engineer on day one. Build modular, move fast, and scale wisely.

Stay tuned

In the next articles, we’ll take a deep dive into the practical setup of each phase — from deploying with Ollama, to scaling up with KServe and Envoy AI Gateway, and finally unlocking advanced MLOps with Kubeflow. You’ll get actionable guides, code snippets, and best practices to implement every stage!

🔗 Links & Resources

• LangChain Official Documentation
• Ollama on GitHub
• KServe Documentation
• KServe Helm Chart
• Kubeflow Official Site
• Kubeflow Pipelines
• OpenTelemetry Docs
• Awesome LLMs (GitHub Curated List)
• Envoy AI Gateway

Feedacks

Have questions ?

Drop your thoughts in the comments !