Building a LanceDB-Powered RAG Chatbot with Streamlit and a Custom Embedding Pipeline

Overview

Large language models are good at generating fluent answers, but they are not naturally grounded in your data. If you ask a model a question about a dataset it has never seen, the answer may sound confident while still being wrong. That gap is exactly where Retrieval-Augmented Generation, or RAG, becomes useful. Instead of asking the model to answer from its general training alone, you first retrieve the most relevant pieces of your own data, then ask the model to answer using that context.

This project is a lightweight Retrieval-Augmented Generation system built around three layers:

1. A document ingestion pipeline that reads files, generates embeddings, and writes records into LanceDB.
2. A retrieval layer that loads rows from LanceDB and finds the most relevant records for a user query.
3. A Streamlit chat interface that sends retrieved context plus the user question to an LLM gateway and renders only the final answer.

The solution is split across two main files:

• enb.py: reads source files, generates embeddings, and stores vectors in LanceDB
• llm.py: loads LanceDB data, retrieves relevant rows, and powers the Streamlit chat application

Architecture At a high level, the system looks like this:

There are two separate runtime paths:

• Offline or pre-processing path:

source file -> embeddings -> LanceDB

• Online inference path:

user question -> LanceDB retrieval -> grounded prompt -> LLM answer

Component Breakdown:

1. Embedding and Storage Layer The ingestion flow lives in enb.py.

Its responsibilities are:

• Read a local input file
• Convert each line or row into text
• Call the embedding endpoint
• Store one LanceDB record per item

The LanceDB target is configured with:

• RAG_DB_DIR = .lancedb
• RAG_TABLE_NAME = uploaded_excel_rows

Each stored record contains fields like:

• doc_id
• source_file
• index
• text
• vector
• indexed_at

This makes LanceDB the persistent vector store for the application.

2. Vector Store Layer The vector store is handled in llm.py through these functions:

• get_lancedb_db()
• get_lancedb_table_by_name()
• get_lancedb_table_names()
• lancedb_rows()

3. Retrieval Layer The retrieval path is implemented in llm.py.

Primary retrieval function:

• search_lancedb_table(…)

How it works:

• Open the selected LanceDB table
• Optionally filter by dataset_id
• Try embedding-based vector search

4. Prompt Construction Layer Prompt grounding is handled by:

• format_rag_context(…)
• build_gateway_messages(…)

The app does not send the user question alone. It builds a grounded prompt with:

• system instructions
• dataset summary
• retrieved row context
• the final user question

This is the core RAG pattern: retrieve first, generate second.

5. Chat UI Layer The frontend is implemented with Streamlit in llm.py.

The Streamlit app provides:

• LanceDB table selection
• optional dataset filter
• model parameters such as temperature and max tokens
• chat input/output
• optional debug mode

In normal mode, the UI is intentionally minimal. It hides:

• preview data
• retrieved rows
• debug payloads

Only the final answer is shown to the user.

Runtime Sequence Here is the request lifecycle for a single user query:

Data Model The current design assumes LanceDB rows can contain some or all of these fields:

• doc_id
• dataset_id
• source_name
• row_number
• text
• payload
• vector
• indexed_at

The app is tolerant of schema variation:

• if payload exists, it is parsed as JSON
• if text is missing, text can be reconstructed from payload fields
• if some export methods are missing, the row loader falls back across multiple LanceDB APIs

That makes the retrieval layer flexible, but it also means consistency in the LanceDB schema will improve reliability.

Why This Design Works This architecture is practical for small and medium internal RAG apps because:

• LanceDB is local and easy to manage
• Streamlit gives a fast UI with minimal code
• embeddings are generated once and reused
• retrieval is separated from generation
• debug mode allows visibility without cluttering the normal UI

It is also operationally simple:

• one local vector database
• one ingestion script
• one chat app

Current Strengths

• Clean split between ingestion and serving
• Works even when LanceDB methods differ by version
• Has fallback search when semantic retrieval fails
• Keeps the user-facing UI minimal
• Uses retrieved context to reduce hallucination risk

Embedding model: text-embedding-3-small (Azure)

LLM model: Amazon Bedrock