Building a Full RAG System with turbovec: The Memory-Efficient Vector Index That Needs No Training

How to build a fast, private, production-ready Retrieval-Augmented Generation pipeline in Python using turbovec — the Rust-powered vector index built on Google Research’s TurboQuant.

If you’ve ever tried to put a Retrieval-Augmented Generation (RAG) system into production, you already know the dirty secret: the demo is easy, but scaling it is brutal. Your embeddings balloon in size, your RAM bill explodes, your vector database needs constant retraining, and somewhere along the way your “simple” search layer becomes the most expensive and fragile part of the entire stack.

Building RAG with turbovec: 16x Vector Compression, Zero Training

This is exactly the problem turbovec was built to solve.

turbovec is a vector index written in Rust with clean Python bindings, built on top of TurboQuant — a quantization algorithm from Google Research published at ICLR 2026. The headline claim is striking: a 10 million document corpus that would consume 31 GB of RAM as float32 fits into roughly 4 GB with turbovec, and it searches faster than FAISS while doing it.

In this article, we’ll cover the what, the why, and the how. By the end, you’ll understand the algorithm that makes turbovec special, and you’ll have a complete, copy-paste-ready RAG pipeline in Python.

What Is turbovec?

At its core, turbovec is the search engine that sits at the heart of any RAG system. When you ask a question, something has to take your query, compare it against millions of stored document embeddings, and return the most relevant matches in milliseconds. That something is a vector index, and turbovec is a particularly clever one.

What sets it apart comes down to four ideas working together.

First, compression. turbovec uses TurboQuant to shrink embeddings by roughly sixteen times. A typical 1536-dimensional OpenAI embedding occupies 6,144 bytes as float32. turbovec stores it in about 384 bytes. Multiply that saving across millions of documents and the economics of running a vector store change completely.

Second, speed. The library ships hand-written SIMD kernels that exploit NEON instructions on ARM chips and AVX-512 on x86. In benchmarks, it outpaces FAISS FastScan by twelve to twenty percent on ARM hardware. You are not trading accuracy for size here — you get both.

Third, no training phase. This is the part engineers fall in love with. Traditional product-quantization approaches require you to train a codebook on representative data before you can index anything, and to retrain when your data distribution drifts. turbovec needs none of that. The quantization is data-oblivious, computed from mathematics rather than from your dataset.

Fourth, privacy by default. turbovec is a pure local library. There is no managed cloud service and no data leaving your machine or your VPC. Pair it with a local embedding model and you have a fully air-gapped retrieval stack — a non-negotiable requirement for teams in healthcare, finance, legal, and defence.

Why Use turbovec for RAG?

Let’s translate those features into the benefits that actually matter when you ship.

It demolishes the RAM wall. The single most common reason RAG projects stall is memory. Embeddings are big, and float32 storage is unforgiving. By quantizing to two-bit or four-bit representations, turbovec lets you serve millions of documents on a single modest machine instead of a fleet of memory-heavy servers. That is the difference between a viable product and an unfundable one.

It keeps your data private. Because nothing is managed and nothing leaves your infrastructure, you can run the entire pipeline inside a regulated environment. Combine turbovec with a local embedding model such as bge, e5, or nomic-embed, and your sensitive documents never touch a third-party API.

It removes an entire category of engineering toil. With no training step, there is no codebook to fit, no parameters to tune per dataset, and no rebuilds as your corpus grows. You add vectors and they are instantly searchable. The reason this works is elegant: after a random rotation, every coordinate of a vector follows a known statistical distribution, regardless of what the original data looked like. Because that distribution is predictable in advance, the optimal quantization buckets can be precomputed with math. The resulting codebook lands within 2.7 times of the Shannon information-theoretic lower bound on distortion — remarkably close to theoretically optimal.

It does hybrid retrieval properly. Most vector stores apply metadata filters after scoring, which means they waste compute scoring documents they then throw away, often hurting recall in the process. turbovec applies filters as an allowlist inside the SIMD kernel. Selective filters — tenant access controls, time windows, keyword candidate sets — avoid most of the search cost rather than paying for it and discarding the result. Filtering becomes free performance instead of a recall tax.

It drops into the tools you already use. turbovec ships integrations for LangChain, LlamaIndex, Haystack, and Agno. In most cases you swap a single import and keep the rest of your pipeline untouched.

How turbovec Works Under the Hood

Before we build, it helps to understand the pipeline that turns a raw vector into a compressed, searchable entry.

The vector first has its magnitude stripped out and stored separately, so the index can reason about direction cleanly. Next comes the crucial step: a random rotation that pushes the coordinates into a known Beta or Gaussian distribution. A calibration stage then shifts and scales each coordinate, and a Lloyd-Max quantizer places the optimal buckets. Finally everything is bit-packed down to two or four bits per coordinate.

At search time, the query is rotated once into the same domain and scored directly against the codebook values using those SIMD kernels. Crucially, the stored database vectors are never decompressed — scoring happens directly on the packed representation, which is why it stays so fast.

The whole flow looks like this:

flowchart LR
    A[Raw vector] --> B[Normalize: strip and store norm]
    B --> C[Random rotation: known distribution]
    C --> D[Calibrate: shift and scale per coordinate]
    D --> E[Lloyd-Max quantize: optimal buckets]
    E --> F[Bit-pack: 2-bit or 4-bit]
    F --> G[(Compressed index)]
    H[Query] --> I[Rotate once] --> J[SIMD score on packed data] --> G

Building a Full RAG System Step by Step

Now the fun part. We’ll build a complete pipeline: embedding, chunking, storage, retrieval with filtering, and generation.

Installation

pip install turbovec
pip install sentence-transformers   # local, air-gapped embeddings

or generation you can use any LLM — a local model through Ollama or llama.cpp, or a hosted API. We’ll keep that layer swappable.

The Embedding Layer

We use a local embedding model so the stack stays private. Note that bge-small produces 384-dimensional vectors; always match your index dimension to your model’s output.

from sentence_transformers import SentenceTransformer
import numpy as np

class Embedder:
    def __init__(self, model_name="BAAI/bge-small-en-v1.5"):
        self.model = SentenceTransformer(model_name)
        self.dim = self.model.get_sentence_embedding_dimension()  # 384

    def encode(self, texts):
        emb = self.model.encode(
            texts, normalize_embeddings=True, convert_to_numpy=True
        )
        return emb.astype(np.float32)   # turbovec expects float32

Chunking

Long documents must be split into overlapping passages so retrieval is precise and context windows stay manageable.

def chunk_text(text, chunk_size=512, overlap=64):
    words = text.split()
    chunks, step = [], chunk_size - overlap
    for i in range(0, len(words), step):
        chunk = " ".join(words[i:i + chunk_size])
        if chunk.strip():
            chunks.append(chunk)
    return chunks

The Vector Store

We build our store on turbovec’s IdMapIndex, which gives stable 64-bit IDs that survive deletions — essential for any real document store where content gets updated or removed. The index holds vectors only, so we keep the chunk text and metadata in a sidecar dictionary.

import json
import numpy as np
from turbovec import IdMapIndex

class TurboVecStore:
    def __init__(self, dim, bit_width=4):
        # bit_width=4 -> 16 buckets per coordinate, higher recall
        # bit_width=2 -> 4 buckets per coordinate, maximum 16x compression
        self.index = IdMapIndex(dim=dim, bit_width=bit_width)
        self.docs = {}
        self._next_id = 0

    def add(self, vectors, texts, metas):
        ids = np.arange(self._next_id, self._next_id + len(texts),
                        dtype=np.uint64)
        self.index.add_with_ids(vectors, ids)
        for i, t, m in zip(ids, texts, metas):
            self.docs[int(i)] = {"text": t, "metadata": m}
        self._next_id += len(texts)
        return ids

    def search(self, query_vec, k=5, allowlist=None):
        q = query_vec.reshape(1, -1).astype(np.float32)
        scores, ids = self.index.search(q, k=k, allowlist=allowlist)
        results = []
        for s, i in zip(scores[0], ids[0]):
            doc = self.docs[int(i)]
            results.append({"id": int(i), "score": float(s),
                            "text": doc["text"], "metadata": doc["metadata"]})
        return results

    def remove(self, doc_id):
        self.index.remove(np.uint64(doc_id))   # O(1) deletion
        self.docs.pop(doc_id, None)

    def save(self, prefix):
        self.index.write(f"{prefix}.tvim")
        with open(f"{prefix}.docs.json", "w") as f:
            json.dump({"docs": self.docs, "next_id": self._next_id}, f)

    @classmethod
    def load(cls, prefix):
        store = cls.__new__(cls)
        store.index = IdMapIndex.load(f"{prefix}.tvim")
        with open(f"{prefix}.docs.json") as f:
            data = json.load(f)
        store.docs = {int(k): v for k, v in data["docs"].items()}
        store._next_id = data["next_id"]
        return store

The Ingestion Pipeline

Here is where turbovec’s online ingest shines. We batch the chunks, embed them, and add them straight into the index. There is no training call and no rebuild — the moment a vector is added, it is searchable.

class RAGIngestor:
    def __init__(self, embedder, store):
        self.embedder = embedder
        self.store = store

    def ingest_documents(self, documents):
        chunks, metas = [], []
        for doc in documents:
            for j, chunk in enumerate(chunk_text(doc["text"])):
                chunks.append(chunk)
                metas.append({
                    "source": doc["source"],
                    "chunk_index": j,
                    "tenant": doc.get("tenant", "default"),
                })
        BATCH = 256
        for i in range(0, len(chunks), BATCH):
            batch = chunks[i:i + BATCH]
            vecs = self.embedder.encode(batch)
            self.store.add(vecs, batch, metas[i:i + BATCH])
        print(f"Ingested {len(chunks)} chunks.")

The Retriever With Hybrid Filtering

This is where we use turbovec’s at-kernel allowlist. When a query is scoped to a single tenant, we build a list of allowed IDs and pass it straight into the search. The filter is applied during scoring, not after it, so there is no recall penalty.

class Retriever:
    def __init__(self, embedder, store):
        self.embedder = embedder
        self.store = store

    def retrieve(self, query, k=5, tenant=None):
        qvec = self.embedder.encode([query])[0]
        allowlist = None
        if tenant:
            ids = [i for i, d in self.store.docs.items()
                   if d["metadata"]["tenant"] == tenant]
            allowlist = np.array(ids, dtype=np.uint64)
        return self.store.search(qvec, k=k, allowlist=allowlist)

The Generation Layer

Retrieval finds the evidence; the language model writes the answer. We build a grounded prompt that forces citations and discourages hallucination, then hand it to whichever model you prefer.

def build_prompt(query, contexts):
    ctx = "\n\n".join(
        f"[{i+1}] (source: {c['metadata']['source']})\n{c['text']}"
        for i, c in enumerate(contexts)
    )
    return f"""Answer the question using ONLY the context below.
Cite your sources with [n]. If the answer is not present, say you don't know.

CONTEXT:
{ctx}

QUESTION: {query}

ANSWER:"""

def generate(prompt):
    # Swap this for any local or hosted LLM.
    import requests
    r = requests.post("http://localhost:11434/api/generate",
                      json={"model": "llama3", "prompt": prompt, "stream": False})
    return r.json()["response"]

Wiring It All Together

class RAGPipeline:
    def __init__(self, bit_width=4):
        self.embedder = Embedder()
        self.store = TurboVecStore(dim=self.embedder.dim, bit_width=bit_width)
        self.ingestor = RAGIngestor(self.embedder, self.store)
        self.retriever = Retriever(self.embedder, self.store)

    def add(self, docs):
        self.ingestor.ingest_documents(docs)

    def save(self, prefix):
        self.store.save(prefix)

    def ask(self, query, k=5, tenant=None):
        contexts = self.retriever.retrieve(query, k=k, tenant=tenant)
        answer = generate(build_prompt(query, contexts))
        return {"answer": answer, "sources": contexts}

if __name__ == "__main__":
    rag = RAGPipeline(bit_width=4)
    rag.add([{
        "source": "handbook.md", "tenant": "acme",
        "text": "turbovec is a Rust vector index built on TurboQuant. "
                "It compresses embeddings roughly sixteen times with no "
                "training phase and searches faster than FAISS."
    }])
    rag.save("my_rag_index")

    result = rag.ask("How much does turbovec compress vectors?", tenant="acme")
    print(result["answer"])
    for s in result["sources"]:
        print(f"  [{s['score']:.3f}] {s['metadata']['source']}")

That is a complete RAG system: ingest, store, retrieve, filter, and generate — backed by a vector index that needs a fraction of the memory and none of the training.

Engineering-Friendly Tips for Production

A few practices will save you pain as you move from prototype to production.

Choose your bit width deliberately. Two-bit quantization gives you maximum compression and is ideal when you are RAM-bound on a very large corpus. Four-bit gives you higher recall and is the safer default for most RAG workloads, where it actually matches or beats FAISS on accuracy and converges to near-perfect recall after just a handful of results.

Always match your index dimension to the exact output of your embedding model, or you will get shape errors at insert time. Prefer IdMapIndex over the plain index for anything real, because stable IDs and constant-time deletes are what let you update documents safely. Remember that the index stores vectors only — persist the sidecar JSON alongside the .tvim file or you will lose your text. And lean on the allowlist for tenant isolation, access control, and time-based filtering; it is one of the few places in engineering where the correct, secure choice is also the faster one.

Finally, you usually do not need to build from source. The pip wheel ships SIMD kernels that detect AVX-512, AVX2, or NEON automatically at runtime, so you get the optimized path for your hardware for free.

Conclusion

RAG is no longer the hard part of building with language models — the retrieval layer is. turbovec attacks the three pains that most often derail these projects: it slashes memory with roughly sixteen-times compression, it eliminates training and rebuild cycles by computing quantization from mathematics rather than your data, and it keeps everything private and local. On top of that it is genuinely fast, and its at-kernel filtering turns hybrid search from a compromise into an advantage.

If you are starting a new RAG project, or you are hitting the RAM wall on an existing one, turbovec is well worth a serious look. Install it, drop in the pipeline above, and point it at your own documents.

You can find the project on GitHub at github.com/RyanCodrai/turbove

If this guide helped you, give it a clap and follow for more hands-on articles on RAG, vector search, and applied machine learning.

Hosting Your turbovec RAG on AWS — The Simplest Possible Way

Step 1 — Launch the Instance

In the AWS Console → EC2 → Launch Instance:

• AMI: Amazon Linux 2023 (Arm64)
• Type: t4g.small
• Storage: 20 GB gp3
• Security group: allow ports 22, 80, 443
• Attach an IAM role with bedrock:InvokeModel permission

That’s the only clicking you do.

Step 2 — Paste This One Script

Expand Advanced details → User data and paste this. It installs everything and starts the server on first boot:

#!/bin/bash
dnf update -y
dnf install -y python3.11 python3.11-pip

pip3.11 install turbovec fastapi "uvicorn[standard]" sentence-transformers boto3

cat > /home/ec2-user/server.py <<'PY'
import boto3, json
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
from turbovec import Index

app = FastAPI()
model = SentenceTransformer("BAAI/bge-small-en-v1.5")
index = Index(dim=384, quantization="4bit")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
docs = {}

class Add(BaseModel):
    id: str; text: str
class Q(BaseModel):
    query: str; k: int = 3

@app.post("/add")
def add(r: Add):
    index.add(r.id, model.encode(r.text)); docs[r.id] = r.text
    return {"count": len(docs)}

@app.post("/search")
def search(q: Q):
    hits = index.search(model.encode(q.query), k=q.k)
    ctx = "\n".join(docs.get(h.id, "") for h in hits)
    body = json.dumps({"anthropic_version":"bedrock-2023-05-31","max_tokens":512,
        "messages":[{"role":"user","content":f"Context:\n{ctx}\n\nQ: {q.query}"}]})
    r = bedrock.invoke_model(modelId="anthropic.claude-3-haiku-20240307-v1:0", body=body)
    return {"answer": json.loads(r["body"].read())["content"][0]["text"]}
PY

# free HTTPS
dnf install -y 'dnf-command(copr)'
dnf copr enable -y @caddy/caddy
dnf install -y caddy
echo 'rag.yourdomain.com {
    reverse_proxy localhost:8000
}' > /etc/caddy/Caddyfile
systemctl enable --now caddy

# run the API as a service
cat > /etc/systemd/system/rag.service <<'EOF'
[Unit]
After=network.target
[Service]
User=ec2-user
WorkingDirectory=/home/ec2-user
ExecStart=/usr/bin/uvicorn server:app --host 0.0.0.0 --port 8000
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now rag

Step 3 — Point Your Domain

Allocate an Elastic IP, attach it to the instance, then add an A record for rag.yourdomain.com pointing to that IP. Caddy grabs a free Let's Encrypt certificate automatically.

If you don’t have a domain, just skip Caddy and hit the box directly on port 8000 over HTTP for testing.

Step 4 — Use It

curl -X POST https://rag.yourdomain.com/add \
  -H "Content-Type: application/json" \
  -d '{"id":"1","text":"turbovec uses 4-bit TurboQuant compression."}'

curl -X POST https://rag.yourdomain.com/search \
  -H "Content-Type: application/json" \
  -d '{"query":"what compression does turbovec use?"}'

That’s the Whole Thing

You now have a live, HTTPS, auto-restarting turbovec RAG API on a single Graviton box for ~$8–10/month (and possibly $0 for the first year on Free Tier).

The flow on every request:

flowchart LR
    A[Query] --> B[Embed]
    B --> C[turbovec search]
    C --> D[Bedrock answer]
    D --> E[Return JSON]

This is the only thing your current setup is missing for long-term safety. Add load-on-start and save-on-change:

import boto3
s3 = boto3.client("s3")
BUCKET, KEY = "my-rag-bucket", "index.tvq"

# On startup: restore the index if a backup exists
try:
    s3.download_file(BUCKET, KEY, "/tmp/index.tvq")
    index = Index.load("/tmp/index.tvq")
except Exception:
    index = Index(dim=384, quantization="4bit")  # fresh start

def persist():
    index.save("/tmp/index.tvq")
    s3.upload_file("/tmp/index.tvq", BUCKET, KEY)

Don’t think “S3 or EC2” — think “EC2 and S3.” EC2 does the thinking, S3 keeps the memory safe. That’s the cheap, fast, long-term-solid answer for turbovec.

Fully Durable turbovec RAG — One Paste, Survives Anything

Here’s your complete bootstrap script with S3 persistence baked in. Same simplicity — one box, one paste — but now the index auto-restores on boot and auto-saves on every change. The box can crash, reboot, or be replaced and your data is safe.

Before You Launch: Create the S3 Bucket

One command from your laptop (or click “Create bucket” in the console):

aws s3 mb s3://my-rag-bucket --region us-east-1

Then make sure the EC2 IAM role can read/write it. Attach this inline policy to the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::my-rag-bucket/*"
    },
    {
      "Effect": "Allow",
      "Action": "bedrock:InvokeModel",
      "Resource": "*"
    }
  ]
}

The Complete User-Data Script

Same EC2 launch as before (Amazon Linux 2023 Arm64, t4g.small, ports 22/80/443, IAM role attached). Paste this into Advanced details → User data:

#!/bin/bash
dnf update -y
dnf install -y python3.11 python3.11-pip

pip3.11 install turbovec fastapi "uvicorn[standard]" sentence-transformers boto3

cat > /home/ec2-user/server.py <<'PY'
import os, json, threading, boto3
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer
from turbovec import Index

BUCKET   = "my-rag-bucket"
IDX_KEY  = "index.tvq"
DOC_KEY  = "docs.json"
IDX_PATH = "/tmp/index.tvq"

app    = FastAPI()
model  = SentenceTransformer("BAAI/bge-small-en-v1.5")
s3     = boto3.client("s3")
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
lock   = threading.Lock()
docs   = {}

# ---------- restore on startup ----------
try:
    s3.download_file(BUCKET, IDX_PATH.split("/")[-1] and IDX_KEY, IDX_PATH)
    index = Index.load(IDX_PATH)
    docs  = json.loads(s3.get_object(Bucket=BUCKET, Key=DOC_KEY)["Body"].read())
    print(f"Restored {len(docs)} docs from S3")
except Exception as e:
    print(f"No backup found, starting fresh ({e})")
    index = Index(dim=384, quantization="4bit")

# ---------- persist helper ----------
def persist():
    index.save(IDX_PATH)
    s3.upload_file(IDX_PATH, BUCKET, IDX_KEY)
    s3.put_object(Bucket=BUCKET, Key=DOC_KEY, Body=json.dumps(docs))

class Add(BaseModel):
    id: str; text: str
class Q(BaseModel):
    query: str; k: int = 3

@app.post("/add")
def add(r: Add):
    with lock:
        index.add(r.id, model.encode(r.text))
        docs[r.id] = r.text
        persist()                      # save to S3 on every change
    return {"count": len(docs)}

@app.post("/search")
def search(q: Q):
    hits = index.search(model.encode(q.query), k=q.k)
    ctx  = "\n".join(docs.get(h.id, "") for h in hits)
    body = json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 512,
        "messages": [{"role": "user",
                      "content": f"Context:\n{ctx}\n\nQ: {q.query}"}]
    })
    r = bedrock.invoke_model(
        modelId="anthropic.claude-3-haiku-20240307-v1:0", body=body)
    return {"answer": json.loads(r["body"].read())["content"][0]["text"]}

@app.get("/health")
def health():
    return {"docs": len(docs)}
PY

# ---------- free auto-HTTPS ----------
dnf install -y 'dnf-command(copr)'
dnf copr enable -y @caddy/caddy
dnf install -y caddy
echo 'rag.yourdomain.com {
    reverse_proxy localhost:8000
}' > /etc/caddy/Caddyfile
systemctl enable --now caddy

# ---------- run API as a service ----------
cat > /etc/systemd/system/rag.service <<'EOF'
[Unit]
After=network.target
[Service]
User=ec2-user
WorkingDirectory=/home/ec2-user
ExecStart=/usr/bin/uvicorn server:app --host 0.0.0.0 --port 8000
Restart=always
[Install]
WantedBy=multi-user.target
EOF
systemctl enable --now rag

The Durability Flow

flowchart TB
    Boot[Box boots] --> Check{Backup<br/>in S3?}
    Check -->|Yes| Restore[Load index + docs from S3]
    Check -->|No| Fresh[Start empty index]
    Restore --> Ready[API ready]
    Fresh --> Ready
    Ready --> Add[POST /add]
    Add --> Save[Save to S3 immediately]
    Save --> Ready

Test It End-to-End

# add a doc — gets persisted to S3
curl -X POST https://rag.yourdomain.com/add \
  -H "Content-Type: application/json" \
  -d '{"id":"1","text":"turbovec uses 4-bit TurboQuant compression to fit large indexes in tiny RAM."}'

# reboot the instance from the console, wait a minute, then:
curl https://rag.yourdomain.com/health
# {"docs": 1}   ← it survived!

curl -X POST https://rag.yourdomain.com/search \
  -H "Content-Type: application/json" \
  -d '{"query":"what compression does turbovec use?"}'

One Tweak for High Write Volume

Saving on every /add is perfect for low/medium write traffic. If you'll be ingesting thousands of docs at once, switch to a debounced timer so you batch saves instead of hitting S3 each time:

import threading
_timer = None
def schedule_persist():
    global _timer
    if _timer: _timer.cancel()
    _timer = threading.Timer(10.0, persist)   # save 10s after last add
    _timer.start()

Then call schedule_persist() instead of persist() inside /add. This collapses a burst of writes into a single S3 upload.

ou’re Done

This is the full package for your Medium post:

• ✅ One paste, single Graviton box
• ✅ Auto-HTTPS via Caddy
• ✅ Auto-restart via systemd
• ✅ Durable — index lives in RAM for speed, mirrored to S3 (11 nines) for safety
• ✅ ~$8–10/mo total

EC2 does the thinking, S3 keeps the memory — exactly the cheap, fast, long-term-solid setup we landed on.