From DevOps to GenAI Cloud Architect # 2: Building Production-Ready Enterprise RAG Systems

Previously, I published my journey from DevOps to GenAI Cloud Architect and explored concepts such as AI Gateways, Embeddings, Vector Databases, and Token Governance.

From DevOps to GenAI Cloud Architect # 1: Building Enterprise AI Systems Beyond Chatbots

One question kept coming back:

“Okay, but how do enterprises actually build Retrieval-Augmented Generation (RAG) systems in production?”

Because let’s be honest.

Most RAG tutorials on the internet look like this:

PDF → Embeddings → Pinecone → OpenAI

And while that helps explain the idea, real enterprise systems are far more complex.

Banks. Healthcare organizations. Telecom providers. Media platforms.

They require governance, observability, security, and reliability.

This article explores what production-ready RAG systems actually look like.

Why RAG Exists

Large Language Models are powerful.

However, they suffer from several limitations:

❌ They don’t know your company’s internal knowledge.

❌ Their training data becomes outdated.

❌ They can hallucinate.

❌ They cannot provide source attribution.

Retrieval-Augmented Generation solves these problems by combining retrieval systems with generative models.

The result? — — — Grounded AI.

Production RAG Architecture

Confluence
GitLab
Jira
PDFs
SharePoint
Runbooks
Logs
       ↓
Data Ingestion
(API Connectors)
       ↓
Document Parsing
(PyPDF / Unstructured / Tika)
       ↓
Chunking
(Recursive / Semantic)
       ↓
Embeddings
(Titan / BGE / OpenAI)
       ↓
OpenSearch Vector Store
(HNSW Index)
       ↓
Retrieval
(Hybrid Search: BM25 + Vector)
       ↓
Reranking
(BGE Reranker / Cohere)
       ↓
Prompt Construction
(LangChain / LlamaIndex)
       ↓
AI Gateway
(Authentication + Governance)
       ↓
LLM Router
(Cost / Capability Routing)
       ↓
Claude / Gemini / Bedrock / GPT
       ↓
Guardrails
(Presidio / Prompt Guard)
       ↓
Answer + Sources
       ↓
Observability
(Langfuse / Grafana)
       ↓
Evaluation
(Ragas / DeepEval)
       ↓
Continuous Improvement

🏗️ Production RAG Architecture

Methods & Technologies Used at Each Stage

Understanding RAG conceptually is easy.

Building it in production is where things become interesting.

Most demonstrations stop at:

PDF → Embeddings → Vector Database → LLM

Real enterprise systems involve significantly more engineering.

Let’s break down the major stages of a production-grade RAG architecture.

Step 1: Knowledge Ingestion

Enterprise knowledge lives everywhere.

Examples include:

• Confluence documentation
• GitLab repositories
• Jira tickets
• Operational runbooks
• Security standards
• PDFs and Word documents
• Internal wiki pages
• SharePoint sites

The first challenge is building pipelines capable of continuously ingesting this information.

The objective is simple:

Keep the AI system synchronized with organizational knowledge.

Common Methods

• API Connectors
• ETL Pipelines
• Event-driven ingestion
• Scheduled synchronization jobs

Enterprise Examples

• Confluence REST APIs
• GitLab APIs
• Jira APIs
• Microsoft Graph APIs
• Kafka
• Amazon S3

Step 2: Chunking — The Hidden Superpower

One of the biggest misconceptions about RAG is assuming that embeddings determine retrieval quality.

In reality:

Good chunking often matters more than good models.

Chunking determines how information is divided before embedding.

Fixed Chunking

Simple.

Fast.

May break context unexpectedly.

Recursive Chunking

Preserves logical document structure.

Most commonly used in production systems.

Examples:

• Headings
• Paragraphs
• Sentences

Semantic Chunking

Groups information based on meaning.

Higher retrieval quality.

Higher computational cost.

Structure-Aware Chunking

Preserves the layout of:

• Markdown documents
• HTML pages
• Tables
• Technical documentation

Enterprise Favorite

✅ Recursive Chunking

Step 3: Embeddings — Converting Language into Mathematics

Computers don’t understand language.

They understand mathematics.

Embeddings transform text into numerical vectors.

Example:

Restart Jenkins Service

↓

[0.21, -0.47, 0.88, ...]

Because embeddings capture semantic meaning:

Restart Jenkins

can match:

Reboot Jenkins

despite using different words.

Popular Embedding Models

• Amazon Titan Embeddings
• BGE Large
• OpenAI Embeddings
• E5 Models
• Gemini Embeddings

Step 4: OpenSearch as a Vector Database

Many engineers know OpenSearch through:

• SIEM workloads
• Observability platforms
• Log analytics
• Security monitoring

Fewer realize that OpenSearch also supports vector search.

Documents are stored alongside their embeddings.

When a user submits a question:

1. The query is embedded.
2. OpenSearch performs similarity search.
3. Relevant chunks are retrieved.

This enables semantic retrieval across enterprise knowledge systems.

Enterprise Advantages

✅ Existing operational expertise

✅ Unified search platform

✅ Security and access controls

✅ Cost efficiency

Step 5: HNSW — The Engine Behind Retrieval

Searching millions of vectors efficiently is difficult.

Comparing every vector against every query isn’t practical.

Modern vector databases rely on Approximate Nearest Neighbor (ANN) algorithms.

The most common is:

HNSW (Hierarchical Navigable Small Worlds)

Benefits include:

✅ High retrieval speed

✅ Excellent accuracy

✅ Scalability

Alternative Approaches

• IVF (Inverted File Index)
• Flat Search (Brute Force)

Enterprise Favorite

✅ HNSW

Step 6: Hybrid Search — The Best of Both Worlds

Semantic search is powerful.

Keyword search remains essential.

Consider these examples.

Query:

INC12345

Requires exact matching.

Query:

How do I restore Kubernetes worker nodes?

Requires semantic understanding.

Hybrid Search combines both approaches:

BM25

+

Vector Search

Benefits

✅ Exact matching

✅ Semantic understanding

✅ Improved recall

Enterprise Favorite

✅ Hybrid Search

Step 7: Reranking — Improving Precision

Initial retrieval often prioritizes recall.

Reranking improves precision.

The process typically looks like this:

Retrieve Top 50

↓

Rerank

↓

Return Top 5

Cross-encoders evaluate the relationship between queries and retrieved chunks more accurately.

Popular Rerankers

• BGE Reranker
• Cohere Rerank
• Jina AI Reranker

The outcome:

✅ Higher precision

✅ Better answers

✅ Reduced hallucinations

Step 8: AI Gateways — The Control Plane of Enterprise AI

As GenAI adoption scales, enterprises face new challenges.

Questions emerge:

• Who manages API keys?
• Which models should applications use?
• How are tokens tracked?
• How do we implement cost controls?
• How do we audit prompts?

AI Gateways solve these problems.

Core Capabilities

✅ Authentication

✅ Model routing

✅ Token governance

✅ Rate limiting

✅ Prompt policies

✅ Audit logging

✅ Cost visibility

AI Gateways are rapidly becoming the control plane of enterprise AI.

Step 9: Observability for AI

DevOps taught us an important lesson:

You cannot improve what you cannot observe.

The same applies to AI systems.

Modern AI platforms require visibility into:

• Retrieval quality
• Hallucination rates
• Token consumption
• Model latency
• Prompt effectiveness
• User satisfaction

This discipline is now known as:

LLMOps

Popular Tooling

• Langfuse
• Arize Phoenix
• Prometheus
• Grafana
• Ragas
• DeepEval

The future of Platform Engineering increasingly includes operating intelligent systems.

What Production RAG Really Looks Like

It’s not:

PDF → Pinecone → OpenAI

It’s this:

Enterprise Knowledge Sources

↓

Knowledge Ingestion

↓

Chunking

↓

Embeddings

↓

OpenSearch (HNSW)

↓

Hybrid Retrieval

↓

Reranking

↓

Prompt Construction

↓

AI Gateway

↓

Claude / Gemini / Bedrock

↓

Observability

↓

Evaluation

↓

Continuous Improvement

The Future of Platform Engineering

Traditional Platform Engineers managed:

• Kubernetes

• Terraform

• CI/CD

• Observability

Tomorrow’s Platform Engineers will additionally manage:

• AI Gateways

• Vector Databases

• RAG Pipelines

• Token Governance

• LLM Observability

• Agentic Workflows

The foundations remain the same.

Automation.

Reliability.

Security.

Platform Thinking.

The systems we operate are simply becoming intelligent.

Final Thoughts

Over the past few months, my own journey has expanded from cloud-native engineering into GenAI infrastructure.

I’ve found that the most successful AI practitioners aren’t abandoning DevOps principles. They’re extending them.

Because building enterprise AI systems isn’t just about choosing an LLM. It’s about building the infrastructure around intelligence.

And I believe this intersection of DevOps, Platform Engineering, and GenAI will define the next generation of cloud architects.

And I believe that may become one of the most important engineering disciplines of the next decade.

If you’re also navigating the transition from DevOps, SRE, or Platform Engineering into GenAI Infrastructure, I’d love to hear what you’re learning and building.

Let’s grow together.

Part 3: From RAG to Agentic AI: The Next Evolution of Platform Engineering (Coming Soon)