Build a Production-Style RAG App on Google Cloud (Vertex AI Gemini + Vector Search)

If you’ve built a “hello world” RAG demo, you already know the next problem: turning it into something reliable—low latency, grounded answers, easy evaluation, and a clean path to deploy.

This Medium post walks through a production-style reference pipeline on Google Cloud, using:

• Vertex AI Gemini for generation
• Vertex AI Text Embeddings (e.g., gemini-embedding-001) for embeddings
• Vertex AI Vector Search for retrieval (Matching Engine)
• Cloud Run to ship an API

What you’ll build

A minimal, production-ready RAG architecture has four stages:

1. Ingest & chunk documents
2. Embed & index chunks into Vertex AI Vector Search
3. Retrieve top-k chunks for each query
4. Generate a grounded answer with Gemini and return citations

Prerequisites

• A Google Cloud project with Vertex AI API enabled

Python environment

• Install the Google Gen AI SDK:

pip install --upgrade google-genai

Vertex AI uses environment variables like these (example from Google docs):

export GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
export GOOGLE_CLOUD_LOCATION="global"
export GOOGLE_GENAI_USE_VERTEXAI=True

Step 1) Ingest and chunk documents (the quiet hero of RAG)

Chunking is the biggest lever in RAG quality. A good default is:

• chunk by headings if possible
• otherwise chunk by paragraphs
• keep overlap (~10–15%) to preserve context boundaries

Store each chunk with metadata:

• doc_id, source (URL/file), section, created_at
• optional: product, region, access_level (for filters later)

This metadata becomes critical for:

• retrieval filters (only “policy” docs)
• access control
• evaluation slicing (“why did the system fail for region=EU?”)

Step 2) Create embeddings with Vertex AI (Gemini embeddings)

Vertex AI’s embeddings API supports models like gemini-embedding-001, and you can set task type and output dimensionality.

Here’s a Python example using the Gen AI SDK:

from google import genai
from google.genai.types import EmbedContentConfig

client = genai.Client()

def embed_texts(texts: list[str]) -> list[list[float]]:
    resp = client.models.embed_content(
        model="gemini-embedding-001",
        contents=texts,
        config=EmbedContentConfig(
            task_type="RETRIEVAL_DOCUMENT",
            output_dimensionality=3072,
            title="MyKnowledgeBase",
        ),
    )
    # resp.embeddings is a list; each item contains a vector
    return [e.values for e in resp.embeddings]

Why the task_type matters: it helps the model optimize embeddings for retrieval scenarios. Why output_dimensionality matters: smaller embeddings reduce storage and can improve speed with minimal quality loss (depending on your domain).

Step 3) Index chunks in Vertex AI Vector Search

At this point you have a list of:

• chunk_id
• embedding vector
• metadata fields (doc_id, source, etc.)

Create a Vector Search index and deploy it to an Index Endpoint (Google’s Vector Search docs cover index types and management).

Tip: choose streaming vs batch updates based on whether you’ll update continuously or periodically.

Step 4) Retrieve top-k chunks (nearest neighbors)

Once you’ve deployed your index, you can query it from Python using the Vertex AI SDK.

from google.cloud import aiplatform

def retrieve(
    project: str,
    location: str,
    index_endpoint_name: str,
    deployed_index_id: str,
    query_embedding: list[float],
    k: int = 5,
):
    aiplatform.init(project=project, location=location)

    endpoint = aiplatform.MatchingEngineIndexEndpoint(
        index_endpoint_name=index_endpoint_name
    )

    neighbors = endpoint.find_neighbors(
        deployed_index_id=deployed_index_id,
        queries=[query_embedding],
        num_neighbors=k,
    )
    return neighbors[0]  # results for the first query

Production tip: add lightweight post-processing:

• drop near-duplicates
• diversify sources (don’t return 5 chunks from the same paragraph)
• enforce metadata filters (if the user isn’t allowed to see certain docs)

Vector Search also supports filtering and diversity mechanisms (“crowding”) — use them early to avoid repetitive context.

Step 5) Generate a grounded answer with Gemini (with citations)

Now you have:

• the user query
• top-k retrieved chunks (text + source)

Use Gemini to answer only from provided context.

Google’s Vertex AI docs show the Gen AI SDK pattern for generate_content and model naming (e.g., gemini-2.5-flash).

from google import genai
from google.genai.types import HttpOptions

client = genai.Client(http_options=HttpOptions(api_version="v1"))

def answer_with_citations(question: str, contexts: list[dict]) -> str:
    # contexts: [{"source": "...", "text": "..."}, ...]
    context_block = "\n\n".join(
        [f"[Source: {c['source']}]\n{c['text']}" for c in contexts]
    )

    prompt = f"""
You are a factual assistant.
Use ONLY the context below. If the context is insufficient, say you don’t have enough information.
Cite sources using the bracketed Source tag.

QUESTION:
{question}

CONTEXT:
{context_block}
""".strip()

    resp = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
    )
    return resp.text

Make it “production-style” (what actually matters)

1) Add an “abstain” rule

If retrieval confidence is weak (low similarity scores), don’t generate. Return:

“I don’t have enough information in the current knowledge base.”

This single rule can cut hallucinations dramatically.

2) Build a tiny evaluation set

Start with 50–200 queries:

• question
• expected doc/source
• expected key facts

Track:

• retrieval recall@k (did we fetch the right doc?)
• groundedness/faithfulness (did the answer stay inside context?)
• latency p50/p95
• abstain rate

3) Cache what’s expensive

Cache:

• query embeddings
• retrieval results for popular questions

Vertex AI also supports context caching patterns for generative apps — useful when you reuse the same long context frequently.

Deploy as a simple API (Cloud Run)

Wrap the pipeline in a single endpoint:

POST /ask:

1. embed query
2. retrieve neighbors
3. generate answer
4. return answer + citations

Cloud Run gives you:

• easy scaling
• low ops overhead
• clean CI/CD

What to publish next (to strengthen a GDE-style portfolio)

If you want this to stand out as a community artifact, publish:

• a GitHub repo with a “reference implementation”
• a short YouTube demo (5–8 minutes)
• a talk deck showing architecture + eval results (even a small benchmark)

Closing

RAG isn’t just “LLM + vector DB.” It’s an engineering system: chunking, metadata, evaluation, guardrails, and deployment discipline.

If you want, I can tailor this Medium post into:

• a part 2 (“Hybrid search + reranking on Vertex AI”)
• a hands-on tutorial with a real repo structure (Cloud Run + IaC)
• a GDE-ready artifact pack (blog + demo + slides outline)

not great; I need descriptive properly formatted

Thought for 15s

Got it — the last one read like documentation notes. Below is a proper Medium-style article: descriptive, story-driven, clean headings, scannable bullets, and “production reality” details (evaluation, guardrails, latency, metadata, security). You can paste this directly into Medium.

Building a Production-Ready RAG System on Google Cloud (Vertex AI Gemini + Vector Search)

Most Retrieval-Augmented Generation (RAG) tutorials stop right after “it works.”

But real users don’t care that your demo works once.

They care that it works fast, stays grounded, handles messy documents, and doesn’t hallucinate when retrieval fails. That’s where most RAG projects break.

This article is a practical blueprint for a production-style RAG system on Google Cloud, using:

• Vertex AI Gemini for generation
• Vertex AI text embeddings for semantic search
• Vertex AI Vector Search (Matching Engine) for fast retrieval
• Cloud Run to deploy an API you can actually ship

What you’ll get from this post

By the end, you’ll know how to design a RAG pipeline that:

• answers using your documents (not guesses)
• includes citations back to sources
• has guardrails when context is weak
• can be evaluated (not “vibes-based”)
• is deployable behind a clean API

The core idea: RAG is a system, not a model

A production RAG pipeline is four systems stitched together:

1. Document ingestion & chunking
2. Embeddings & indexing
3. Retrieval & context assembly
4. Generation + safety rules

When RAG fails, it’s usually because one of these isn’t engineered.

Architecture (clean mental model)

Think in two flows:

Offline flow (batch)

• collect documents → clean text → chunk → embed → push vectors to index

Online flow (per user query)

• query → embed → retrieve top-k → build prompt → Gemini → answer + citations

Google Cloud components

• Cloud Storage: raw docs + parsed text + chunk JSONL
• Vertex AI Embeddings: convert chunks into vectors
• Vertex AI Vector Search: retrieve top-k context under low latency
• Vertex AI Gemini: produce grounded response
• Cloud Run: serve the /ask endpoint
• Cloud Logging: request tracing + debugging

Step 1: Ingestion and chunking (the #1 driver of RAG quality)

Most teams underestimate chunking. In practice, chunking is more important than the LLM for retrieval quality.

A good default chunking strategy

• Aim for 300–800 tokens per chunk
• Use 10–15% overlap so context isn’t cut mid-thought
• Prefer splitting by headings/sections over arbitrary token slicing
• Keep chunks consistent (wild chunk sizes hurt retrieval)

Store metadata with every chunk

This is what makes RAG controllable:

• doc_id, title, source_url or filename
• section / heading
• created_at
• optional: product, region, language, access_level

Why metadata matters:

• Filtering: “only policy docs” or “region=US”
• Permissions: only show context user can access
• Evaluation: analyze failures by doc type or topic

If you only store vectors without metadata, you’ll regret it later.

Step 2: Create embeddings using Vertex AI

You embed two things:

• document chunks (offline)
• user queries (online)

A clean practice is to use a retrieval-tuned embedding configuration:

• documents embed as “retrieval documents”
• queries embed as “retrieval queries”

This improves retrieval alignment.

Practical rule: Use the same embedding model for both document chunks and queries.

Step 3: Index the vectors in Vertex AI Vector Search

Vector Search gives you managed, scalable nearest-neighbor retrieval without running your own vector DB.

Index design tips

• Keep vectors + metadata together
• Choose top-k (commonly 5–15)

Plan for updates:

• batch re-index nightly/weekly (simpler)
• streaming updates (if docs change frequently)

Retrieval pitfalls (common in real systems)

• top-k results are duplicates from the same source section
• a single long doc dominates retrieval
• irrelevant “similar” chunks win due to generic wording

Fixes:

• deduplicate similar chunks
• diversify by source
• use metadata filtering
• later: add a reranker (optional but powerful)

Step 4: Retrieval that supports evaluation (not just output)

When the user asks a question, don’t only return an answer.

Log the retrieval behavior too:

• retrieved sources
• similarity scores
• which chunk ids were used
• what filters were applied

This is how you debug:

• “Gemini hallucinated” usually means retrieval was weak
• “Answer is wrong” often means retrieval fetched the wrong doc
• “Answer is slow” often means retrieval was fine but generation prompt was bloated

Step 5: Prompt Gemini to stay grounded + provide citations

Your prompt must force disciplined behavior.

A good grounded answer policy

• Use ONLY the provided context
• If context is insufficient → say so
• Cite sources

Example prompt pattern (clean + effective)

System

• “You are a factual assistant. Do not invent facts. Use only the provided context.”

Developer

• “Answer the user question strictly from context. Provide citations as [source] per paragraph.”

User

• question + context blocks

Output format (recommended)

Return something structured:

• answer
• citations (list of source ids)
• confidence (simple heuristic based on retrieval scores)

This makes your app easier to integrate into UI and easier to evaluate.

Production guardrails (this is what separates demos from real systems)

Guardrail 1: “No context → no generation”

If top similarity score is below a threshold, do not call Gemini.

Return:“I don’t have enough information in the knowledge base to answer that.”

This one rule reduces hallucinations dramatically.

Guardrail 2: Keep prompts small

Stuffing 15 chunks into the context kills latency and increases confusion.

Start with:

• top 5–8 chunks
• trim to the most relevant and diverse

Guardrail 3: Safe fallbacks

If Gemini fails or times out:

• return citations + retrieved context snippets
• log failure + request id

Evaluation (how you prove RAG works)

RAG quality should be measured like a search system + a generation system.

Minimum evaluation set

Build a set of 50–200 test queries:

• query text
• expected source doc(s)
• expected key facts (short bullets)

Metrics that matter in practice

Retrieval

• recall@k: did we retrieve the right doc?
• duplicate rate: how redundant are results?

Generation

• groundedness: is the answer supported by context?
• abstain rate: did we correctly refuse when context is weak?

Performance

• latency p50 / p95
• cost per request (embedding + retrieval + generation)

Even a small evaluation set is better than guessing.

Deployment on Cloud Run (simple API you can ship)

Wrap the online pipeline in a single endpoint:

POST /ask

1. embed query
2. retrieve top-k
3. build grounded prompt
4. call Gemini
5. return answer + citations

Recommended additions:

• request id in every response
• structured logs including retrieval results
• basic rate limiting
• caching for frequent queries (query embedding + retrieval results)

Common mistakes (and how to avoid them)

Mistake 1: RAG without citations

If you don’t return citations, users won’t trust your system.

Mistake 2: No “abstain” behavior

Most hallucinations happen when retrieval fails. Handle that explicitly.

Mistake 3: No metadata strategy

Without metadata, you cannot filter, secure, or debug.

Mistake 4: No evaluation loop

If you can’t measure retrieval quality, you can’t improve it.