Background picture by Panumas Nikhomkhai.

Limits of RAG and implications for self-hosted AI

Why scalability isn’t infinite, and Retrieval-Augmented Generation not a panacea

I. A NEW BUZZWORD?

RAG (Retrieval-Augmented Generation) is a new buzzword in the field of LLMs (Large Language Models), itself a rather limited part of AI (Artificial Intelligence), which should invite a need for clarification, analytics and nuance. This is what this article will first focus on, before deriving some implications for self-hosted AI, or local AI.

Local AI is something I already discussed in another article [1], specifically some implementation. Like with llama.cpp in a Python venv, used in CLI, therefore mostly consisting of text generation and configured through a file. Or with Easy Diffusion to use models and configurations for self-hosted generative purposes, such as for generating pictures. This article had also mentioned a few facts about sovereignty and compliance, two notions at the interplay of the tech industry and politics.

Large Language Models work by predicting the next token based on a prompt. But an LLM is made out of several compounds: the tokenizer, the word embeddings, the weighted neural network, the temperature setting, an optional retriever system… RAG as a technology is being discussed in papers, for several reasons.

Figure 1. Diagram of self-hosted RAG. Made with Canva, May 6th, 2026. “Limits of RAG and implications for self-hosted AI”, Emilia Lilith-Lolita Hoarfrost, Medium. A client (CLI or browser) is able to communicate with a self-hosted LLM (typically llama.cpp?), that connects to a RAG local database according to specified modalities, using FastAPI (Python), leading to better LLM results that are curated to a specific use, for instance compliance to local laws, better medical results, or otherwise applications specific to a given vector/industry.

• A key feature of ChatGPT, one of the pioneers in the rise of popular consciousness of Large Language Models and derivated applications (How many SaaS are actually wrappers for an AI under-the-hood API? Is this industrial model even sustainable?), was its training, done on a large corpus including dated events. Meaning being aware of the hyper-contemporary was one of its limitations, limiting its uses in understanding the news, like in geopolitics.
• The accrued industrial investments (mentioned in the prior article [1]) mean greater competition, hence the will for LLM-as-a-Service companies (OpenAI, Alphabet, Meta, xAI…) to counter known weaknesses of LLMs (statistical parroting, hallucinations, outdated knowledge), to up the service. R&D is exactly that: industrial investments, applied research.
• Early market pioneered by Perplexity, which to my knowledge was one of the first LLM-as-a-Service cloud products to actually link to pertinent sources, and check the web in real time.

II. WHAT IS RETRIEVAL-AUGMENTED GENERATION?

But what exactly is RAG? According to a 2024 IBM video with IBM Senior Research Scientist Marina Danilevsky [2]:

“In the RAG framework, the generative model actually has an instruction that says: “No, no, no.”, “First, go and retrieve relevant content.” “Combine that with the user’s question and only then generate the answer.””

It can mean that the LLM checks online, but also that there is a potential database that is given priority in polishing answers by the LLM. For instance, Grok, as an LLM aligned to further a far-right agenda, can also treat emerging topics, hence the usefulness of its integration to the X ecosystem as a social media.

There’s also Grokipedia, an xAI attempt at competing against Wikipedia that explicitly (it’s in the name) refers to Grok, since it’s considered Wikipedia is left-aligned in its Overton window. Notably because it allows decentralized contributions to entries, and particularly valorizes scientific or academic publications as sources. This shows novelty in the idea of sustainable use of artificial intelligence, but future researchers, academics and historians will need to put emphasis on the biased nature of Grokipedia. There should also be an interesting field of study across ethics, computer science, artificial intelligence and epistemology in studying the discrepancies between Grokipedia and Wikipedia.

The example of Grokipedia hints at some of the problems with RAG: alignment, how it’s weighted… Admitting news, blog articles, social media posts are trusted as a first-hand account, RAG will be hurt by the contradictory nature of several voices. Which is supposedly accounted for as part of LLM trainings. In 2020, Google had to issue a patch and an official apology [3] after users found out that the labels inferred about pictures (possibly as part of semantic SEO or GEO [5]) had racist biases, like more easily associating black people with guns…

“ “Our investigation found some objects were mis-labeled as firearms and these results existed across a range of skin tones. We have adjusted the confidence scores to more accurately return labels when a firearm is in a photograph.” Ms Frey added that *Google* had found “no evidence of systemic bias related to skin tone.*” ” [quote of a quote, [3], but one has to read between the lines with corporate talk and damage control]*

But if we start to hierarchize sources, which is explicitly the goal with some RAG implementations (notably when you want to ensure compliance with specific policies documented in a database that you indicate to dominate in the fact-checking step of the prompt generation by an LLM), the new problem becomes that of distinguishing the value of sources. Well, for scientists, scholars and jurists it’s actually always been there… Let’s say that your RAG database has policies on State laws, and then on federal laws in the US, or French laws, and European laws for French politics surrounding tech. Which one should dominate? Maybe jurists can come up with answers that differ according to when they were educated, where they were educated…

But that’s perhaps not a problem with LLMs using Retrieval-Augmented Generation, and a problem with legal gray areas. The hierarchy of sources in law is already a thing everyone is implicitly, or at times explicitly, aware of. So if the expert-system is parroting its sources, and you’re given the authority on choosing those sources, which happens not just with a formal RAG implementation, but also when in the situation of an LLM discussion you give a few bits of context in prompting and further answers to the LLM, it should still reflect hierarchy of sources. In the tech industry that may be about tech documentations, or formalizations of languages, like C99…

That’s why AI-assisted coding is easier when the agent has access to your codebase and its different subdirectories and files, rather than isolated snippets. How many times have I had problems due to a scattered nature of my files? Resorting to using a single HTML file to vibe-code a quick web application… But this faces the problem of tokens, and in the case of local/self-hosted AI, you get to specify tokenization alongside batch sizes.

There is a difference between a system prompt and Retrieval-Augmented Generation, since a system prompt, which can be customized in ChatGPT or llama.cpp alike, is actually a condition prior to even the first answer by an LLM, whereas there are three detailed steps in RAG [2].

III. HOW TO IMPLEMENT RAG IN LOCAL AI?

It actually means two types of implementations: either online access; or a local (at the level of the self-hosted implementation) access to data as context. Indeed, admitting you’re using *Azure, AWS, Google Cloud* or some other cloud service provider, you can still be using a more private virtualization, that ensures compliance policies requirements, including cryptography. Usually that’s a more “premium” offer **of the cloud service provider.

And then you delineate the modalities of the three steps [2]:

• The storage of information that is to be prioritized.
• The way information is retrieved.
• A query-conditioned selection.

[embed]“Build Your Own Self-Hosted AI Chatbot with RAG | End-to-End Tutorial”, TechieTalksAI, YouTube, September 17th, 2025. [6]

This video has a detailed architecture for its own guide, which is not totally self-hosted and offline.

1. The front-end is a browser client (HTML/CSS/JS). On this client is an embed for the chatbot.
2. There’s an Nginx front-end (static site + proxy).
3. The proxy goes to a *FastAPI *(Python) back-end (AI chatbot + RAG**).
4. This proxy communicates with OpenAI API (for GPT + embeddings).
5. Then a markdown (text format) documentation (RAG knowledge) is implemented with a vector store.

It’s not what we want because there’s a point of failure: ChatGPT, not to mention you need an API key and that can lead to transfer of personal information. There are limits to the sovereignty, but we can use this idea for the implementation of a local RAG and AI.

Indeed, local AI can use browser clients, it’s something done in networking for maintenance anyways, and permits a great amount of interoperability. Easy Diffusion does that, instead of stuff like Discord to access Midjourney.

Because FastAPI is Python we can build the infrastructure locally. But for the chatbot embeddings, we can actually use *llama.cpp* and make up our own configuration. But the guide has code notably for building out the vector store [6] so maybe you’d do well by following it. It’s not really the focus of the article, to guide you through implementation, **though.

IV. HOW MUCH INFORMATION? INFINITELY SCALABLE?

But too much information kills the quality of information. And this is what a study [4] demonstrated.

From the abstract:

“Moving away from the usual RAG evaluation with short answers, we explore the more challenging long-form question answering in two domains, where a good answer has to utilize the entire context. Our findings indicate that final QA performance improves steadily with up to 15 snippets but stagnates or declines beyond that. Finally, we show that different general-purpose LLMs excel in the biomedical domain than the encyclopedic one, and that open-domain evidence retrieval in large corpora is challenging.”

And in the paper, notably the final section on limitations of the study:

“Our study is limited to two datasets, thus making it possible that some findings would not universally generalize to different domains and tasks. Additionally, we only evaluate the models in a zero-shot setting, whereas a few-shot setting [closer to actual use of an LLM] with some examples of questions and answers would have led to a more uniform performance across models.”

One of my own questions remains: whatever is the size of a snippet? But the paper has a more detailed breakdown that I’m not able to understand.

What remains, however, is that too much context is not okay and this is why there is often only a dozen kept by Perplexity. This probably has implications for not giving too long and contradictory contexts for local AIs built to use a RAG verification step process.

As a consequence, more context is not always better, and thus the myth of scalability for LLM efficiency in hitting benchmarks, is perhaps off-the-tables for anyone serious in the game. Despite this, however, I’m sure you’ll find customers that are like, going to throw massive amounts of storage at the issues.

V. CONCLUDING: RAG FOR SELF-HOSTED AI

We’ve seen through part I that RAG was apparently the new buzzword in the AI hype cycle that everyone was now talking about. What it means is just the normal human habit of… checking your sources, and choosing the best ones. Part II was more about developing on what it meant specifically, and some of the problems such as a hierarchy of sources. Part III was more dedicated to the specificities of implementation for local AI. Part IV discussed the findings of a study [4] on the actual greater limit of how much information pulled was really useful, giving indications that scalability is not endless… and with GEO being a new thing, marketers are going to try to game the system and maybe this will lower the use of online RAG due to an endless stream of AI-generated content. Overall, RAG is an interesting lockstep to fight some of the earliest problems of LLMs, such as hallucination and dated information in the corpora trained on. It remains to be seen how it does evolve and how tech policies will follow up on that. We know that legal adaptation can take a while compared to the newer fields that digital humanities concern themselves with.

VI. REFERENCES

[1] Article. “Can Resilient AI Survive the Ouroboros? — Green IT, AI Bubble and Data Centers: On Defending Local AI Models”, Emilia Lilith-Lolita Hoarfrost, Medium, February 6th, 2026. https://medium.com/@emiliahoarfrost/can-resilient-ai-survive-the-ouroboros-green-it-ai-bubble-and-data-centers-defending-local-ai-4f464e8a98bc

[2] Video. “What is Retrieval-Augmented Generation (RAG)? (6:32)”, Senior Research Scientist Marina Danilevsky, IBM, September 7th, 2024. https://www.ibm.com/think/videos/rag

[3] Article. “Google apologizes after its Vision AI produced racist results”, Dr. Nicolas Kayser-Bril, Algorithm Watch, April 7th, 2020. https://algorithmwatch.org/en/google-vision-racism/

[4] Study. “On the Influence of Context Size and Model Choice in Retrieval-Augmented Generation Systems”, Juraj Vladika, Florian Matthes, Findings of the Association for Computational Linguistics: NAACL 2025, February 20th, 2025. https://arxiv.org/abs/2502.14759

[5] Article. “💻📊80 SEO Tips & Tools That Boost Traffic for Indie Blogs, E-Shops & SaaS”, Emilia Lilith-Lolita Hoarfrost, Medium, April 26th, 2026. https://medium.com/@emiliahoarfrost/80-seo-tips-tools-that-boost-traffic-for-indie-blogs-e-shops-saas-59841020d46d

[6] Video. “Build Your Own Self-Hosted AI Chatbot with RAG | End-to-End Tutorial”, TechieTalksAI, YouTube, September 17th, 2025. https://www.youtube.com/watch?v=P01qugIdmJI