From Kurals to Conversations: Building a Vectorless RAG Engine for Thirukkural

Introducing

If you have recently built a Retrieval-Augmented Generation (RAG) system, you probably followed the standard playbook of chunking your data, passing it through an embedding model like OpenAI’s, and storing those dense vectors in a database like Pinecone or ChromaDB for similarity searches. But what if dense embeddings are actually the wrong tool for your specific dataset?

When I set out to build a RAG API for the Thirukkural a classical Tamil text composed of 1330 precise, two-line aphorisms I quickly realized that traditional vector embeddings fumbled the ball, struggling to capture exact phrasing, deep cultural nuances, and the text’s rigid taxonomy. Furthermore, paying for OpenAI API calls and hosting a heavy vector database for a passion project felt like total overkill, which led me to pivot to a “Vectorless RAG” architecture. By swapping out complex dense embeddings for a lightning-fast, keyword-smart BM25 algorithm (which ranks documents based on exact terms) and replacing paid models with a free, open-source Mistral LLM via Hugging Face, I built a zero-cost, highly accurate RAG API that respects the exact structure of ancient literature without the unnecessary infrastructure.

Unlocking the Wisdom of Thirukkural

Table of Contents

1. Introducing
2. What is Vectorless RAG? And Why Do We Need It?
3. The Challenge: Parsing Classical Taxonomies
4. Architecture Overview
5. Building the BM25 Retrieval Engine
6. Integrating the Free Hugging Face LLM
7. FastAPI as the Operational Glue
8. Deploying Live on Hugging Face Spaces
9. The Big Takeaway
10. Production API Reference: The 23 Endpoints 1. Discovery & Structure 2. Document Lookup 3. Vectorless Search 4. RAG Generation 5. Conversational Memory 6. Analytics & Telemetry 7. Export & Sharing 8. System & Admin
11. Ready to Explore the API?

What is Vectorless RAG? And Why Do We Need It?

Traditional RAG relies heavily on semantic search, where an embedding model translates your text into a mathematical representation of its general “meaning.” While this works beautifully for broad, modern questions, it often falls flat when dealing with domain-specific vocabulary like archaic words, translated poetry, or deep cultural terms or even when you need exact keyword matching, sometimes returning a conceptually similar answer that is factually incorrect.

• Domain-Specific Vocabulary: Embeddings often misunderstand archaic words, specific cultural terms, or translated poetry.
• Exact Keyword Matching: If a user searches for a highly specific term, semantic search might return a conceptually similar but factually incorrect document.

“Vectorless” RAG fixes this by taking a step back to classical information retrieval, swapping out heavy mathematical vectors for an algorithm called BM25 (Best Matching 25), which acts like a supercharged keyword search. By tracking how often a specific word appears in a single document compared to the entire dataset, BM25 focuses on exact, high-priority terms rather than vague concepts. For a structured, precise text like the Thirukkural, where a user is usually hunting for a highly specific phrase or an exact philosophical concept, this keyword-first approach isn’t just lighter and completely free to run, it is actually far more accurate than using vectors.

Vectorless RAG (PageIndex)

The Challenge: Parsing Classical Taxonomies

The Thirukkural isn’t just a list of quotes; it has a rigid hierarchy:

• 3 Sections (Paal)
• 133 Chapters (Adhikaram)
• 1330 Couplets (Kurals)

If a user asks, “What does Thiruvalluvar say about the importance of speaking kind words?”, the engine shouldn’t just grab random text. It needs to pinpoint the exact chapter (“The Utterance of Pleasant Words”) and pull the exact translated couplets to feed to the LLM.

Because our data is statically structured in JSON files (details.json, data.json, synonyms.json), we can load it directly into memory. No Pinecone required.

Architecture Overview

To keep things modular and incredibly easy to scale, I broke the system down into four clear, bite-sized layers that each handle one specific job:

• The Data Layer: The entire text sits safely in static JSON files, giving us a lightning-fast, zero-maintenance database without the hassle of spinning up a heavy server.
• The Retrieval Engine: We pair LlamaIndex with the rank-bm25 package to create an ultra-lean indexing system that fetches the exact relevant couplets in milliseconds using pure keyword intelligence.
• The LLM Layer: This layer taps into Hugging Face’s completely free Serverless Inference API to run the highly capable Mistral-7B-Instruct-v0.2 model, giving us smart text synthesis with absolutely zero cloud hosting costs.
• The Routing Layer: FastAPI acts as the brain of the operation, using Pydantic to instantly validate user inputs and smoothly orchestrate the data flow from the initial search query to the final AI-generated response.

Building the BM25 Retrieval Engine

• Instead of setting up a heavy vector database, we use LlamaIndex’s built-in BM25Retriever. At startup, it tokenizes our JSON documents and holds a lightweight search index right in memory. Because the dataset is so small (just 1330 text blocks), this initialization takes only milliseconds and requires zero external infrastructure.

from typing import List, Dict, Any
from rank_bm25 import BM25Okapi
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.schema import NodeWithScore, TextNode, QueryBundle
from app.services.data_manager import data_manager
from app.services.tokenizer import tamil_tokenizer

class VectorlessThirukkuralRetriever(BaseRetriever):
    def __init__(self):
        self.nodes: List[TextNode] = []
        self.corpus_tokens: List[List[str]] = []
        self.bm25: BM25Okapi = None
        self.inverted_index: Dict[str, set] = {}
        self.rebuild_index()
        super().__init__()

    def rebuild_index(self):
        """Hot-reloads data frames and rebuilds memory vectorless index weights from scratch."""
        self.nodes = []
        self.corpus_tokens = []
        self.inverted_index = {}

        for kural in data_manager.kural_data:
            k_num = kural["Number"]
            meta = data_manager.document_tree.get(k_num, {})

            # Consolidate text corpus space for maximum multi-lingual contextual match probability
            searchable_text = (
                f"{kural.get('Line1','')} {kural.get('Line2','')} "
                f"{kural.get('mv','')} {kural.get('sp','')} {kural.get('mk','')} "
                f"{kural.get('Translation','')} {kural.get('explanation','')}"
            )

            node = TextNode(
                text=searchable_text,
                metadata={
                    "kural_number": k_num,
                    "Line1": kural.get("Line1", ""),
                    "Line2": kural.get("Line2", ""),
                    "Translation": kural.get("Translation", ""),
                    "section": meta.get("section", "Unknown"),
                    "chapterGroup": meta.get("chapterGroup", "Unknown"),
                    "chapter": meta.get("chapter", "Unknown"),
                    "chapter_translation": meta.get("chapter_translation", "Unknown")
                }
            )
            self.nodes.append(node)
            self.corpus_tokens.append(tamil_tokenizer.tokenize_and_expand(searchable_text))

            # Build Inverted Metadata Lookups for rapid categorical filtering
            for field in ["section", "chapterGroup", "chapter"]:
                val = meta.get(field)
                if val:
                    if val not in self.inverted_index:
                        self.inverted_index[val] = set()
                    self.inverted_index[val].add(k_num)

        self.bm25 = BM25Okapi(self.corpus_tokens)

    def search(self, query_str: str, filters: dict = None, top_k: int = 3) -> List[NodeWithScore]:
            """Direct search method bypassing QueryBundle limitations."""
            filters = filters or {}
            query_tokens = tamil_tokenizer.tokenize_and_expand(query_str)
            if not query_tokens or not self.bm25:
                return []

            bm25_scores = self.bm25.get_scores(query_tokens)

            allowed_kural_ids = None
            for filter_key in ["section", "chapter"]:
                filter_val = filters.get(filter_key)
                if filter_val:
                    matching_ids = self.inverted_index.get(filter_val, set())
                    if allowed_kural_ids is None:
                        allowed_kural_ids = matching_ids.copy()
                    else:
                        allowed_kural_ids = allowed_kural_ids.intersection(matching_ids)

            scored_nodes = []
            for idx, score in enumerate(bm25_scores):
                node = self.nodes[idx]
                k_id = node.metadata["kural_number"]

                if allowed_kural_ids is not None and k_id not in allowed_kural_ids:
                    continue

                if score > 0.0:  
                    scored_nodes.append(NodeWithScore(node=node, score=score))

            scored_nodes.sort(key=lambda x: x.score, reverse=True)
            return scored_nodes[:top_k]

    def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
            """Fallback for standard LlamaIndex internal pipelines."""
            return self.search(query_str=query_bundle.query_str)

vectorless_retriever = VectorlessThirukkuralRetriever()

Integrating the Free Hugging Face LLM

To keep this entirely cost-free, we bypass paid OpenAI credits completely. By using Hugging Face’s free Serverless Inference API, which integrates seamlessly with LlamaIndex we can route our context-stuffed prompts straight to Mistral using a standard HF_TOKEN, giving us capable LLM intelligence with zero hosting overhead.

import os
from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI

hf_token = os.getenv("HF_TOKEN")

llm = HuggingFaceInferenceAPI(
    model_name="mistralai/Mistral-7B-Instruct-v0.2",
    api_key=hf_token,
    temperature=0.2,
    max_tokens=512
)

FastAPI as the Operational Glue

FastAPI ties the entire system together. When a user hits the /api/v1/rag/ask endpoint with a question, the routing layer instantly queries the BM25 engine for the top 3 most relevant couplets. It then injects those exact couplets into a structured prompt template and hands it off to Mistral to generate a culturally accurate, contextual answer.

import os
from fastapi import APIRouter, Depends
from llama_index.core import get_response_synthesizer
from llama_index.core import Settings as LlamaSettings
from llama_index.llms.huggingface_api import HuggingFaceInferenceAPI

from app.models.schemas_rag import RAGRequest, RAGResponse, SearchResultNode
from app.services.index_engine import vectorless_retriever
from app.api.deps import get_api_key
from app.core.config import settings

router = APIRouter()

# 1. Instantiate the Free Hugging Face LLM
LlamaSettings.llm = HuggingFaceInferenceAPI(
    model_name=settings.LLM_MODEL,
    token=settings.HF_TOKEN,
    temperature=settings.LLM_TEMPERATURE,
    max_tokens=settings.LLM_MAX_TOKENS
)

# 2. Instantiate a standalone LLM Synthesizer
response_synthesizer = get_response_synthesizer()

@router.post("/ask", response_model=RAGResponse, dependencies=[Depends(get_api_key)])
def ask_rag(request: RAGRequest):
    """Main RAG Endpoint: Applies Vectorless search context to the Free HF LLM."""
    filters = request.filters.model_dump(exclude_none=True) if request.filters else {}

    # Step A: Retrieve matching nodes
    nodes = vectorless_retriever.search(query_str=request.question, filters=filters, top_k=request.top_k)

    # Step B: Synthesize the response using Mistral/LLaMA via the Inference API
    response = response_synthesizer.synthesize(query=request.question, nodes=nodes)

    # Step C: Format the output payload
    sources = [
        SearchResultNode(kural_number=n.metadata["kural_number"], text=n.text, score=n.score, metadata=n.metadata)
        for n in response.source_nodes
    ]

    return RAGResponse(question=request.question, answer=str(response), sources=sources)

Deploying Live on Hugging Face Spaces

The final step is getting the API off your local machine. Because this architecture requires absolutely no external databases, the entire project easily fits inside a lightweight Docker container. I deployed it to the free Docker tier on Hugging Face Spaces by writing a simple Dockerfile using a python:3.11-slim image, creating a non-root user to satisfy Hugging Face’s security standards, and safely hiding the HF_TOKEN in the Space's UI Secrets. The entire API was live and globally available in less than three minutes.

The Big Takeaway

As developers, we have been conditioned to think that building production-ready AI applications requires a massive, heavy infrastructure stack — complete with costly vector databases, complex chunking strategies, and expensive API keys. But by pairing classical, time-tested algorithms like BM25 with highly capable open-source models like Mistral, you can build lean, blindingly fast, and completely free RAG systems that actually outperform semantic vector search on structured, literature-heavy datasets.

Production API Reference: The 23 Endpoints

To keep this system highly modular and easy for anyone to consume, the backend exposes 23 micro-endpoints split cleanly across 8 core domains. This means frontend devs, data scientists, or downstream apps can query specific layers of the engine independently.

1. Discovery & Structure

Explore the structural breakdown and classical taxonomy of the text.

• GET /api/v1/discovery/structure — Returns the complete global taxonomy map of the text.
• GET /api/v1/discovery/sections — Fetches details for the three macro Sections (Paal).
• GET /api/v1/discovery/sections/{section_name}/chapters — Retrieves all Chapter groups and chapters within a specific section.
• GET /api/v1/discovery/chapters/{chapter_name} — Resolves isolated metadata for a single target chapter.

2. Document Lookup

Perform high-speed direct structural reads, completely bypassing the retrieval network.

• GET /api/v1/lookup/kural/{kural_number} — Fetches the exact text, translations, and commentaries for an isolated couplet.
• GET /api/v1/lookup/chapters/{chapter_name}/kurals — Returns all ten couplets linked to a specific chapter.
• GET /api/v1/lookup/kural/random/daily — An automated utility endpoint generating a curated random couplet daily.

3. Vectorless Search

Interact directly with the keyword-frequency engine and synonym dictionary.

• GET /api/v1/search/raw — Executes a standard unweighted term match across the BM25 index.
• POST /api/v1/search/filtered — Performs an advanced query scoped strictly by specified sections or chapters.
• POST /api/v1/search/tokenize — A debug utility revealing how the underlying custom tokenizer parses sentences.
• GET /api/v1/search/synonyms — Pulls down the current active word-mapping state.
• POST /api/v1/search/synonyms — Hot-updates or appends new definitions to the active synonym dictionary.

4. RAG Generation

The primary endpoint connecting keyword retrieval to large language model synthesis.

• POST /api/v1/rag/ask — Takes a natural language query, orchestrates context gathering through the BM25 layout, assembles the prompt pipeline, and surfaces a validated structural response.

5. Conversational Memory

Manages stateful, multi-turn interactions over stateless cloud sessions.

• POST /api/v1/chat/session/start — Instantiates a unique session tracker and reserves state memory buffers.
• POST /api/v1/chat/send — Routes multi-turn conversational follow-ups into the session-aware context engine.
• GET /api/v1/chat/session/{session_id}/history — Exposes the recorded dialogue logs for a given active session.

6. Analytics & Telemetry

In-memory log processors analyzing traffic metrics and data usage.

• GET /api/v1/analytics/popular-topics — Examines query patterns to aggregate trending concepts.
• GET /api/v1/analytics/stats — Monitors hardware cache states, indexing delays, and overall processing load.

7. Export & Sharing

Format data payloads for external, cross-platform presentation layers.

• GET /api/v1/export/kural/{kural_number}/export — Serializes a highly formatted card object containing structural keys, metadata, and complete translation segments for quick card generation.

8. System & Admin

Core diagnostics for system administrators.

• GET /api/v1/system/health — Verifies index presence and engine uptime. A standard ping yields:

{
  "status": "active",
  "message": "Thirukkural Vectorless RAG API is fully operational.",
  "index_size": 1330
}

• POST /api/v1/system/admin/index/reload — Forces the data layer to re-read JSON sets and rebuild the BM25 inverted mapping state without taking down the active server loop.

Ready to Explore the API?

The complete source codebase, deployment configuration profiles, and interactive playground are fully open-source and live on the web:

• Live App (Interactive Doc-Hugging Spaces): thirukkural-vectorless