Building a self-hosted genAI platform

To reach AI at scale in a company, you quickly realise you need more than an LLM endpoint. You need a platform that accelerates delivery by giving teams reusable building blocks, optimises cost/perf, and secures usage across many teams and use cases. Most components of such a platform are readily available on the hyperscalers. However, for security or sovereign concerns mostly, many companies are building outside hyperscalers (e.g. on a sovereign cloud or on-premise), where setting up such architecture becomes much more challenging.

Self-Hosted GenAI Platform Architecture

This article focuses on the building blocks of a self-hosted genAI platform, with examples of self-hostable technos and learnings from deploying them. The architecture schema above gives an overview of the platform.

PS1: this article does not cover the “tech-generic” (e.g. CICD, RBAC, …) or “traditional ML” (e.g. model registry, …) building blocks although they are naturally important in such platform. PS2: the set of technos listed in this article is non-exhaustive

1. Processing Unit (e.g. GPU)

The hardest part of a self-hosted genAI platform is often simply getting enough powerful GPUs to run the company’s LLM-based use cases. You typically need 4–8 H200s (roughly $35k/H200 in acquisition cost) to run a SOTA model with >100B parameters, to both fit the weights and the KV cache from users’ requests. As the load increases, more replicas and hence more GPUs will be required.

Deploying smaller models to reduce acquisition costs can be considered, but you will most likely end up spending significantly more on engineering labor, as use cases will require many more iterations to reach production. Moreover, if deploying a ChatGPT-like tool for your users, the experience will be disappointing, driving “shadow AI” among your company’s employees. Likewise, “cheaper” GPUs like A100 or L40S are not always a win: if optimally used, H200s have a much lower cost per token than A100s (source) and are much faster than L40S (source).

New inference-first hardware designed specifically for transformer inference, like Groq’s LPUs and Cerebras’ WSE, can reach very high throughput (tokens/s). In some benchmarks, Groq reports 10–100× faster token generation than other providers using Nvidia GPUs. That said, LPUs are less proven than GPUs and come with trade-offs: the stack is more closed at the low level (no CUDA equivalent), not integrated with common serving stacks like vLLM so you don’t get the same out-of-the-box tooling to tune how inference is run for your workload (e.g. trading off throughput vs latency), and it requires many chips in parallel driving up initial CapEx (source). Still, the threat/opportunity to Nvidia remains real, as they “acqui-hired” Groq’s leadership team and licensed their technology for $20B in December 2025.

2. Model Serving

Dedicated LLM serving tools optimise resource usage during generation and expose models through an API. They come in two complementary layers: the single-model serving engine and the distributed serving layer.

The single-model serving engine optimises how one model runs on one (or a few) GPUs. **vLLM has become the default standard thanks to its easy deployment, production-oriented features like continuous batching, and PagedAttention to optimise KV-cache memory management. ZML, a newer competitor entering the inference market, is rebuilding the stack in Zig for tighter control over control flow and memory allocation. They are currently building their inference server, [llmd](https://hub.docker.com/r/zmlai/llmd)**, on top of their framework. Their core idea is that attention is often sparse, so GPUs’ dense matrix multiplications waste compute: instead, they model attention as a graph computation that can run on CPU at speeds comparable to GPU inference. By moving attention and the KV cache to CPU, GPUs mainly store weights (not KV), drastically reducing GPU needs and costs (details & demo in this video). However, ZML has mostly showcased demos so far and has not been used in production at scale to our knowledge.

The distributed serving layer makes inference scalable by orchestrating fleets of single-model engines across a GPU cluster, to run multiple models, autoscale based on real-time demand, and share LoRA adapters and KV cache across instances without replication. Typical providers include **llm-d (not to be confused with ZML’s [llmd](https://hub.docker.com/r/zmlai/llmd)) and [Aibrix](https://github.com/vllm-project/aibrix)** (within the vLLM-project org). Although these products are less than a year old, ByteDance (the team behind Aibrix) reported successful deployments across several business use cases.

3. Guardrails

Guardrails improve the security and safety of genAI systems by detecting and preventing risky behavior such as data exfiltration (e.g. sensitive data leakage through prompt injection) and harmful or policy-violating inputs/outputs. Indeed, even though most modern LLMs are fine-tuned to behave safely, they still fail often enough that additional guardrails are needed.

Most attacks require semantic understanding of the content; as a result, BERT-style classifiers trained for moderation/injection detection are often used because they offer a good trade-off between speed/cost and performance. That said, these classifiers are less nuanced than LLMs, so they tend to produce false positives when toxic vocabulary is present even if the prompt is not harmful. Typical providers include **Azure AI Content Safety (AAICS), which can run on-prem for $200k/year; [Lakera](https://www.lakera.ai/), with a similar pricing range and known for their Gandalf game (which enabled them to collect millions of prompts to train their classifier); and [Guardrails AI](https://www.guardrailsai.com/)**, which is open-source. We benchmarked these solutions in early 2025 (≈1k samples from BeaverTails, PurpleLlama prompt injections, plus custom data): AAICS and Lakera achieved >80% recall with >95% precision (the misses were mostly genuinely ambiguous), while Guardrails AI was far behind (≈20% recall).

Finally, Lakera, Guardrails AI and other solutions like **NeMo Guardrails** enable to run complementary checks such as PII detection or custom filters (e.g. block documents if a “confidential” footer is detected).

PS: We focused here on text guardrails, since they cover the vast majority of needs. Image, audio, and video guardrails are less common, although both NeMo and Azure Content Safety offer some features for image moderation.

4. Vector Database

Within a genAI platform, a vector database generally fulfills two key functions:

• Shared embeddings layer: a central place to store embeddings of the company’s information system so they can be reused across multiple use cases. However, access control, so each user only retrieves documents they are allowed to see, becomes a challenge in complex organisations, that can be addressed with metadata filtering and sharding/partitioning.
• Project-specific space: dedicated capacity for teams that generate and manage their own embeddings independently.

For such a platform, it is best to use a dedicated vector DB (e.g. **Qdrant or [Weaviate](https://weaviate.io/)) over a general-purpose DB with vector capabilities (e.g. pgvector**). Indeed, the former tends to be more optimised for retrieval-heavy workloads and ships more search-related capabilities like hybrid search, faceting, and pre-filtering that will likely be needed, especially if supporting many heterogeneous use cases.

If you want more detail, the comparison I wrote in 2023 still mostly holds. In practice, we still use Qdrant for its performance, feature set, and ease of use.

5. MCPs

The Model Context Protocol (MCP) offers another way to connect LLMs to your company’s information system (or external providers), especially for agentic use cases. Compared to the “shared embeddings layer” (see Vector Database section), using MCP avoids heavy pre-indexing and ingestion pipelines. Instead, the LLM can infer the logical steps to access your data (e.g., generating a SQL query may give better results than vector search in some cases). Just like with a shared embeddings layer, security and access control are challenging. A pattern that’s becoming popular for MCP is the **OAuth 2.0 On-Behalf-Of (OBO) protocol**: it lets a tool call backend services on behalf of the signed-in user, and securely propagates identity when other agents or MCPs are called.

As MCP adoption grows, more products ship MCP integrations out of the box, but it is sometimes necessary to build your own MCP server: either because no connector exists, or because custom logic is needed (e.g. specific domain knowledge or security process). To do so, the **official MCP project provides FastMCP in Python to build an MCP server in just a few lines of code. Projects with specific focus are emerging too, like Google’s [GenAI Toolbox](https://github.com/googleapis/genai-toolbox)**, which enables to easily to deploy an MCP connected to a database with the necessary security and observability layers.

6. AI Gateway

An AI gateway acts as the platform’s single entry point to all AI building blocks: models, vector DBs, MCP tools, etc. It lets you issue dedicated API keys per user/app to track usage and cost, enforce quotas and restrictions (e.g. max spend/consumption), and standardise security and compliance controls like authentication, applying the same guardrails across endpoints, and maintaining an audit trail. Beyond governance, one of the key values is resource pooling: multiple teams can safely share the same model endpoints and retrieval infrastructure, which is especially important for GPU-intensive LLMs where serving capacity is expensive and needs to be amortised.

**LiteLLM** quickly became a go-to AI gateway by unifying many LLM providers behind an OpenAI-compatible API and being very simple to install & use. In practice, some instabilities have been observed, mostly memory leaks that become problematic at scale: several fixes from recent releases tend to mitigate them but they still seem to remain. Hence, good alternatives could be:

• **Kong**, a general-purpose API gateway which was widely deployed in production before the genAI wave (and hence more mature) and has since added AI-related features covering a similar scope as LiteLLM.
• **TensorZero**, an AI gateway implemented in Rust to optimise memory and speed (LiteLLM is in Python)

7. AI Engineering Platform (Monitoring, Observability, Evaluation, …)

AI Engineering Platforms provide the tooling layer to ship genAI use cases with best practices. Tools like **Langfuse or [Basalt](https://getbasalt.ai/)** offer largely comparable capabilities. The differences are often in UX, plus a few minor but practical functionalities. Their most useful features are for observability & monitoring, but they go beyond that. Their core features typically include:

• Observability, to inspect user traces end-to-end and drill into each step (retrieval, reranking, generation, tool calls, etc.) to understand failure modes and derive concrete improvement ideas.
• Monitoring, to track usage, latency, and cost per interaction. This complements, rather than replaces, traditional monitoring like Grafana.
• Evaluation, to run evals on production traces or curated datasets using either manual review or LLM-as-a-judge. While using the latter for strict “true/false” correctness is often unreliable, it is useful to extract high-level signals at scale (e.g. to detect negative signals from users repeatedly rephrasing, showing frustration, or even insulting the AI).
• Prompt versioning. It is most valuable when prompts are updated dynamically at runtime. Otherwise, prompt versioning in code is often sufficient.

In a genAI platform context, this layer has two big benefits: it gives product teams a standard SDK/tooling so they build the reflex to log & analyse everything from day 1 (even in beta), and it provides admins with an overview of platform usage across teams and use cases.

8. No/Low-Code Workflow & Agent Builder

No/low-code workflow & agent builders are commonly used for internal, low-scale use cases and POCs: they speed up delivery and broaden the set of people who can ship automation (beyond just engineers). However, these tools often remain hard to use in practice for most non-technical users.

The space is extremely competitive, with many overlapping tools. **n8n has become the most popular options: it is open source, has a large catalog of integrations, and offers broad functionality, making it suitable for building a wide range of automations, with or without AI. AI-first tools like [Dify](https://dify.ai/)** have also emerged: Dify offers a more genAI-oriented UX and features, but is less general-purpose than n8n when you need more complex automation.

9. Off-the-shelf AI Platforms

Off-the-shelf AI platforms bundle several of the components described above (and you can deploy the others individually), then add the extra layers, e.g. security, governance and UX, to make the platform easily usable by teams. From a cost perspective, the licensing fee is often lower than the true cost of building and maintaining an equivalent stack in-house.

If your infrastructure is already based on OpenShift, **Red Hat OpenShift AI** (RHOAI) is often the go-to option. It provides many of the capabilities you would expect from mature cloud AI platforms (like Vertex AI or Snowflake) to build and deploy use cases: assigning compute to users, experiment tracking, model serving, pipeline scheduling, and more.

On the other hand, platforms such as **LightOn or [Craft AI](https://www.craft.ai/)** are more genAI-specific and focus on providing ready-made use cases based pre-built RAG pipelines and agents, but usually don’t offer the same breadth of foundational “platform building blocks” as RHOAI.

Conclusion

Assembling a ready-to-use genAI platform is both challenging and necessary if you want to scale AI use cases in a self-deployed environment. The landscape is crowded with component, many of them very recent, widely adopted in prototypes, but not always enterprise-mature when it comes to operability, security, and long-term maintenance.

At Theodo, we help companies facing these constraints design and build scalable genAI platforms. If this resonates with your context, don’t hesitate to reach out.