Designing a Self-Hosted RAG System with Open-Source Tools

What This Article Covers

Every answer a RAG system gives is only as good as what happened before the LLM ran - how documents were parsed, chunked, embedded, and indexed.

The standard pipeline

PDF → chunks → embeddings → vector DB → chatbot

A production system adds seven requirements:

parse the document reliably

preserve its structure

chunk it to match how people ask questions

embed every chunk consistently

retrieve with access control

cite the source of each answer

reindex or delete the document later

This article covers each one, on CPU, using open-source components:

• Docling — PDF parsing, OCR, tables, Markdown extraction
• llama-server — CPU embeddings (GGUF models: bge-small, bge-m3, nomic-embed, Qwen3-Embedding)
• Qdrant — vector storage and filtered retrieval
• BM25 — keyword retrieval (via Qdrant sparse vectors or a companion index)
• Cross-encoder reranker — GGUF or ONNX, also on CPU
• Redis + RQ / Arq / Dramatiq — job queue
• Postgres — document state

Architecture

                         ┌──────────────────────────┐
                         │        Upload API        │
                         │ auth / limits / metadata │
                         └────────────┬─────────────┘
                                      ▼
                         ┌──────────────────────────┐
                         │   Document State Store   │
                         │ uploaded / parsing / ... │
                         └────────────┬─────────────┘
                                      ▼
                         ┌──────────────────────────┐
                         │        Job Queue         │
                         │ backpressure / retries   │
                         └────────────┬─────────────┘
                                      ▼
┌──────────────────────────────────────────────────────────────────────┐
│                            Ingest Worker                             │
│                                                                      │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────────────┐    │
│  │ Parse PDF    │───▶│ Chunk by     │───▶│ Embed in batches     │    │
│  │              │    │ structure    │    │                      │    │
│  └──────┬───────┘    └──────────────┘    └──────────┬───────────┘    │
│         │                                           │                │
└─────────┼───────────────────────────────────────────┼────────────────┘
          │ HTTP                                      │ HTTP
          ▼                                           ▼
┌──────────────────┐                       ┌──────────────────────┐
│ Docling          │                       │ llama-server         │
│ parser           │                       │ embeddings (CPU)     │
└──────────────────┘                       └──────────────────────┘

                         worker writes indexed chunks
                                      │
                                      ▼
┌──────────────────────────────────────────────────────────────────────┐
│                               Qdrant                                 │
│ dense vectors + sparse vectors + payload                             │
│ tenant_id / document_id / page range / chunk type                    │
│ parser version / embedding model / index version / text              │
└──────────────────────────────────────────────────────────────────────┘

                                QUERY PATH

                         ┌──────────────────────────┐
                         │     Retrieval Layer      │
                         │ hybrid → rerank → cite   │
                         └────────────┬─────────────┘
                                      │ reads
                                      ▼
                         ┌──────────────────────────┐
                         │          Qdrant          │
                         └────────────┬─────────────┘
                                      │ context
                                      ▼
                         ┌──────────────────────────┐
                         │    Answer With Sources   │
                         └──────────────────────────┘

The ingest side (parse → chunk → embed → index) determines most of the answer quality. Retrieval and generation operate on whatever evidence ingest produced.

1. Parsing

PDFs vary: digital text, scanned images, mixed pages, tables, formulas, rotated or encrypted files, structurally broken pages. They all share the same .pdf extension.

Parsing runs as an isolated service, separate from the API:

API → queue → worker → Docling (isolated)

If the parser crashes or spikes memory, it stays contained. The API continues serving.

The parser returns text plus an ingest report:

pages_total
pages_empty_after_parse
tables_detected
parse_seconds
parser_version
warnings

The report makes parse quality visible before indexing. A document is not indexed until its parse quality is known.

2. Chunking

Fixed-size vs. structure-aware

A fixed-size chunker splits text at a character count (e.g. 1,000 chars with 100 overlap). It works for linear prose.

Technical documents are structured: headings, tables, code blocks, warnings, formulas, captions. A structure-aware chunker preserves these units.

Steps:

1. Detect structure
2. Group atomic units
3. Fit the embedding model’s max input length
4. Mark splits explicitly

Chunks are tokenized with the model’s own tokenizer (limits in section 3).

Atomic units

Unit Purpose Heading + section body topic stays attached to content Table + caption rows, units, labels stay together Code block code stays self-contained Equation block formula stays with context Warning or note operationally critical content stays intact Figure caption visual evidence stays traceable

Size targets

target:      300–600 tokens
working max: ~800–1,200 tokens
hard ceiling: model's max input length

A complete table retrieves better than five fragments of it. A heading attached to its paragraph carries more context than a chunk without one.

Fragment marking

When a unit must be split, the split is marked:

{
  "chunk_type": "table",
  "is_fragment": true,
  "fragment_index": 2,
  "fragment_count": 5
}

Retrieval, the UI, and the evaluator all use this marker.

Context prefix

Each chunk gets a prefix from the parsed structure — document title and heading path:

[Manual: Inverter X / Section 3.2 Operating Limits]
The maximum operating temperature is 60°C ...

Cost: string concatenation from data Docling already produced. Benefit: improved recall on short or ambiguous chunks that would otherwise embed without context.

3. Embedding

CPU embeddings are slower per token than GPU. They work well for asynchronous ingest.

Batching

Embedding requests group 32–128 chunks per call. llama-server, text-embeddings-inference, and infinity all accept batches over HTTP.

Dimension validation

On startup, the worker probes the embedding server and compares the returned dimension to the collection’s dimension. A mismatch stops the worker before incompatible vectors enter the index.

probe embedding server
read vector dimension
compare to collection dimension
halt on mismatch

Versioning

Every chunk stores:

embedding_model
embedding_dimension
embedding_server_image_or_version
index_version

Changing the embedding model is an index migration. Existing and new vectors are not comparable. index_version tracks which generation each vector belongs to.

Model choices

Model Dim Max input Notes bge-small-en-v1.5 384 512 Fast, English bge-m3 1024 8192 Multilingual, dense + sparse + multivector nomic-embed-text-v1.5 768 (Matryoshka) 8192 Truncatable dimensions Qwen3-Embedding-0.6B 1024 32k Multilingual, larger, CPU-runnable

Matryoshka models store full-dimension vectors and allow search against a truncated view (e.g. 256 dims) for speed, with rescoring on the full vector.

4. Metadata

Each vector carries a payload:

{
  "tenant_id": "tenant_123",
  "document_id": "doc_456",
  "document_version": "v2",
  "index_version": "rag_v3",
  "chunk_index": 17,
  "chunk_type": "table",
  "is_fragment": false,
  "page_start": 8,
  "page_end": 9,
  "parser": "docling@2.x",
  "chunker": "structure_v2",
  "embedding_model": "bge-m3",
  "embedding_dim": 1024,
  "content_hash": "sha256:...",
  "text": "..."
}

The payload enables five operations:

1. Access control. Every query filters by tenant_id:

client.search(
    collection_name="docs",
    query_vector=qvec,
    query_filter=Filter(must=[FieldCondition(key="tenant_id", match=MatchValue(value=current_tenant))]),
)

2. Citations. Page ranges and document IDs link each answer to its source.

3. Debugging. The payload traces a retrieved chunk back to its document, page, parser, chunker, and model version.

4. Deletion. Removing a document deletes its vectors by document_id and tenant_id.

5. Reindexing. index_version separates old and new generations during migration.

Deduplication

content_hash catches repeated boilerplate (headers, footers, disclaimers). Duplicates are dropped before embedding.

5. Queued Ingest

A queue sits between the API and the worker. It provides backpressure, retry control, concurrency limits, job visibility, and failure isolation.

Redis + RQ, Arq, or Dramatiq handles this on a CPU stack.

Each document moves through a state machine:

UPLOADED
   │
   ▼
PARSING ──▶ PARSE_FAILED
   │
   ▼
CHUNKING ──▶ SUSPICIOUS_PARSE
   │
   ▼
EMBEDDING ──▶ EMBEDDING_FAILED
   │
   ▼
INDEXING ──▶ INDEXING_FAILED
   │
   ▼
INDEXED

The state machine is the source of truth. Documents in EMBEDDING are not searchable. PARSE_FAILED and SUSPICIOUS_PARSE are surfaced. Operators query the state store to find stuck documents.

6. Retrieval

Dense embeddings match meaning. They are weaker on exact identifiers: part numbers, clause numbers, error codes, serial numbers, acronyms, dates, names, legal references.

BM25 matches exact tokens. Combining both - hybrid retrieval - covers both modes.

                   ┌──────────────────┐
                   │    User Query    │
                   └─────────┬────────┘
              ┌──────────────┴──────────────┐
              ▼                             ▼
     ┌─────────────────┐           ┌──────────────────┐
     │ Dense Retrieval │           │ Keyword (BM25)   │
     └────────┬────────┘           └────────┬─────────┘
              └──────────────┬──────────────┘
                             ▼
                    ┌────────────────┐
                    │ Fusion (RRF)   │
                    └────────┬───────┘
                             ▼
                    ┌────────────────┐
                    │ Cross-encoder  │
                    │ rerank top ~50 │
                    └────────┬───────┘
                             ▼
                    ┌────────────────┐
                    │ Answer + cite  │
                    └────────────────┘

Fusion

Reciprocal Rank Fusion (RRF) merges the dense and sparse rankings:

score(doc) = Σ 1 / (k + rank_i(doc))    k ≈ 60

RRF requires no per-corpus tuning.

Reranking

A cross-encoder scores each candidate against the query directly. It takes the top ~50 from fusion and selects the top 5–8 for the prompt. Open-source CPU rerankers: bge-reranker-v2-m3, mxbai-rerank, ms-marco-MiniLM.

Query transformation

The query is rewritten before search:

• Multi-query — a small LLM produces 2–3 paraphrases; results are fused
• HyDE — the LLM generates a hypothetical answer; that text is embedded and searched
• Decomposition — a multi-part question splits into sub-queries

A 3B–8B model on the same host handles all three.

7. Evaluation

The evaluation set is 20–50 real questions per corpus, each paired with expected evidence:

{
  "question": "What is the maximum operating temperature of inverter X?",
  "expected_document": "inverter_x_datasheet",
  "expected_pages": [4],
  "must_find": ["operating temperature", "60°C"]
}

The primary metric is retrieval accuracy:

Metric Measures Recall@5 evidence found in top 5 Recall@10 evidence found in top 10 MRR rank of first correct chunk empty-page rate parsing failures fragmented-chunk rate chunking damage citation coverage answers linked to sources faithfulness answer uses only retrieved evidence

Ragas, TruLens, or a custom script all work. They call the retriever directly and run on CPU.

The loop runs after any change to: parser settings, chunk logic, embedding model, fusion weights, reranker, or query transformation.

8. Deployment

The full stack runs on a single host:

┌─────────────────────────────────────────────────────────────────┐
│                         Single Host                             │
│                                                                 │
│  ┌─────────────┐       ┌─────────────┐       ┌─────────────┐    │
│  │ API (FastAPI│──────▶│ Redis Queue │──────▶│ Worker(s)   │    │
│  └──────┬──────┘       └─────────────┘       └──────┬──────┘    │
│         ▼                                            ▼          │
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────┐ │
│  │ Postgres    │  │ Docling Svc │  │ llama-server│  │ Reranker│ │
│  │ states/docs │  │ PDF parsing │  │ embeddings  │  │ ONNX    │ │
│  └─────────────┘  └─────────────┘  └──────┬──────┘  └────┬────┘ │
│                                           ▼              ▼      │
│                                       ┌───────────────────┐     │
│                                       │      Qdrant       │     │
│                                       │ dense + sparse    │     │
│                                       └───────────────────┘     │
└─────────────────────────────────────────────────────────────────┘

Upload is asynchronous (queue → worker). Question answering is synchronous (retrieve → rerank → answer).

CPU tuning:

• Qdrant scalar quantization (or binary) bounds RAM
• Payload index on tenant_id, document_id, index_version keeps filtered search fast
• llama-server thread count set explicitly — over-subscription reduces throughput
• Embedding cache on content_hash makes re-uploads free

9. Priority Order

parse > chunk > retrieve > generate

Answer quality
        │
        ▼
depends on evidence quality
        │
        ▼
depends on retrieval quality
        │
        ▼
depends on chunk quality
        │
        ▼
depends on parse quality

10. Checklist

For every document:
  know parse quality
  preserve structure
  build evidence-sized chunks with context prefixes
  deduplicate by content hash
  store page ranges and chunk type
  embed with a known, versioned model
  validate vector dimension on startup
  index with tenant + document metadata

For every query:
  optionally rewrite the query
  retrieve with dense + BM25 fusion
  filter by tenant and permissions
  rerank with a cross-encoder
  answer only from the reranked evidence
  cite the source pages
  measure retrieval against a real question set

RAG quality is decided before generation.