Before Scaling RAG: Ensuring Technical Terms Are Safe

Transitioning to a local glossary guard, regression tests, and an agentic web ingestion design to reduce local LLM predictions

**[If you cannot access the full article, please click here]**

Image generated using AI (ChatGPT) by the author.

Hello,

I’ve been working on ( My lovely Local LLM) Asuli, my own local knowledge system, for a while now. My goal was simple at first:

Read EPUB
Read PDF
Search Medium articles
Ask a question
Get an answer

But as the system grew, I realized this: RAG alone isn’t enough. Because the problem isn’t always “Is the information there?” Sometimes the problem is this: The model misinterprets the correct term. It misapplies an abbreviation to the wrong context. It speaks with certainty about a topic it’s unsure of. It misses the local context. Example: What is OCP?

A general model might interpret this differently in different contexts. But in my context, OCP stands for: OpenShift Container Platform In the context of telco, OpenShift, PaaS, Kubernetes, and platform operations, misinterpreting this term means a serious quality issue. That’s why I developed a new, small layer at Asuli: Technical Glossary Guard

Simply put:

Don't leave certain critical technical terms to the LLM.
First, check the local, verified dictionary.
If there's a match, provide a deterministic response.
If there's no match, continue with the normal flow.

Problem In RAG systems, the following workflow is generally considered: The user asks a question → the system finds the relevant chunks → the LLM generates a response This is a good start. But for some questions, retrieval isn’t even necessary. For example:

What is SRE?
What is RAG?
What is MCP?
What is CKA?
What is OCP?
What is PaaS?
What is SLO?
What is Kafka?

These questions are short but risky. Because:

• Acronyms can vary depending on context
• LLMs may respond with general information
• The specific project context may be lost
• An incorrect expansion can erode trust in RAG’s answers That’s why I first added a small, controlled, and tested technical glossary layer.

Goal My goal was not to completely eliminate the LLM. The goal was this: To prevent the LLM from making predictions where it isn’t necessary. That is: SRE → Site Reliability Engineering RAG → Retrieval-Augmented Generation MCP → Model Context Protocol CKA → Certified Kubernetes Administrator OCP → OpenShift Container Platform For Asuli, these are no longer terms to be “predicted,” but rather “known” terms.

Initial Check First, I checked to see if these terms were present in the current state:

cd /home/ali/projects/telegram
grep -nE '"(sre|rag|mcp|cka|ocp)"' app.py scripts/regression_guard_tests.py || true
git status --short
Then, I took a physical backup before making any changes.
cd /home/ali/projects/telegram
BK="backups/before_technical_glossary_change_$(date +%Y%m%d_%H%M%S)"
mkdir -p "$BK"
cp app.py "$BK"/
cp scripts/regression_guard_tests.py "$BK"/
echo "BACKUP_OK: $BK"

This is an important rule for me: Just committing to Git isn’t enough. Take a physical backup before making any changes to a live system.

Glossary Guard Logic I created a small glossary structure within Asuli. Simplified version:

_TECHNICAL_GLOSSARY = {
 "sre": "...",
 "rag": "...",
 "mcp": "...",
 "cka": "...",
 "ocp": "...",
}

Then I used the following flow for user messages:

1. Private identity guard
2. Public person guard
3. Technical glossary guard
4. Normal chat / RAG flow

This order is important. Because the private person guard must run before the technical glossary. Example: Who is Ali Elmalı? This question is not technical,

This question is not technical; it pertains to the private identity scope. Therefore, the private identity guard should take precedence over the technical dictionary.

Phase 1: Key Acronyms In the first phase, I identified the following terms: SRE RAG MCP CKA OCP I tested:

cd /home/ali/projects/telegram
./venv/bin/python -m py_compile app.py scripts/regression_guard_tests.py
./venv/bin/python scripts/regression_guard_tests.py

Output:

PASS test_technical_glossary_guard_sre
PASS test_technical_glossary_guard_rag
PASS test_technical_glossary_guard_mcp
PASS test_technical_glossary_guard_cka
PASS test_technical_glossary_guard_ocp
ALL_REGRESSION_GUARD_TESTS_PASSED

The purpose of regression testing here is as follows: I added something new. Have the previous correct behaviors been compromised? For example, when adding OCP, the test caught the following: The OCP response should not include an alternative expansion with incorrect context. This may seem minor, but it’s very valuable. Because what often corrupts the RAG system isn’t a major error, but rather a small yet persistent contextual error.

Live Test I restarted the service:

#cd /home/ali/projects/telegram
sudo systemctl restart asuli-bot.service
sleep 3
systemctl status asuli-bot.service --no-pager | sed -n '1,35p'
Log:journalctl -u asuli-bot.service -n 240 --no-pager \
| grep -E "TECHNICAL_GLOSSARY|PERSONAL_IDENTITY|PUBLIC_PERSON|ERROR|Traceback|Unhandled" || true

Expected log:

TECHNICAL_GLOSSARY_GUARD_ANSWERED
Unexpected log:
ERROR
Traceback
Unhandled
Live test examples:
What is OCP?
What is CKA?
What is MCP?
What is RAG?
What is SRE?
What is Kubernetes?

Result:

Result:

OCP → glossary guard
CKA → glossary guard
MCP → glossary guard
RAG → glossary guard
SRE → glossary guard
Kubernetes → normal flow

This distinction is important. Because if the guard is too aggressive, it will catch everything and slow down the system. Here’s what I want: Only catch the critical terms you know. Let general questions you don’t know go through the normal flow.

Phase 2: Cloud-SRE Batch In the second phase, I added the terms “cloud” and “SRE”: PaaS IaaS SLO SLI error budget incident response These are particularly important for operations and platform quality. One of the sample answers:

An SLO is a target value that defines a service’s reliability or performance goal in a measurable way. For SLI: An SLI is a reliability or performance metric that measures a service’s actual behavior. For error budget: An error budget is not a monetary budget, but a reliability tolerance. These answers are brief but contextually appropriate.

Regression test: PASS test_technical_glossary_guard_cloud_sre_batch ALL_REGRESSION_GUARD_TESTS_PASSED

Phase 3: Platform-Data Batch In the third phase, I added the terms “platform” and “data layer”: OpenShift Ceph Kafka PostgreSQL / Postgres MongoDB / Mongo These may be general technology terms. But in my context, they have more specific meanings. Example for Kafka: Kafka is a distributed event streaming platform used to process high-volume event and message streams.

Asuli context:

• topic, partition, producer, consumer, and consumer group
• asynchronous communication between microservices
• event-driven architecture
• monitoring of throughput, retention, replication, and lag For Ceph: Ceph is a distributed open-source storage platform that offers object, block, and file storage capabilities.

For PostgreSQL: It is a candidate for a managed database in the context of DbaaS and platform services. For MongoDB: It is a document-based NoSQL database. Live quiz: What is Kafka? What is OpenShift? What is Ceph? What is PostgreSQL? What is Postgres? What is MongoDB? What is Mongo? What is Kubernetes?

Result:

Kafka → dictionary protection
OpenShift → dictionary protection
Ceph → dictionary protection
PostgreSQL/Postgres → dictionary protection
MongoDB/Mongo → dictionary protection
Kubernetes → normal flow

Roadmap Entry I didn’t just leave each phase in the code. I also incorporated them into the Roadmap:

sed -n '/Technical glossary flow:/,/Roadmap governance:/p' docs/ROADMAP_INDEX.md

Output summary:

Technical glossary updates:

- SRE glossary guard added for Site Reliability Engineering
- RAG glossary guard added for Retrieval-Augmented Generation
- MCP glossary guard added for Model Context Protocol
- CKA glossary guard added for Certified Kubernetes Administrator
- OCP glossary guard added for OpenShift Container Platform
- Cloud-SRE glossary batch added for PaaS, IaaS, SLO, SLI, error budget, incident response
- Platform-Data glossary batch added for OpenShift, Ceph, Kafka, PostgreSQL/Postgres,...
Git log:
git --no-pager log --oneline --decorate -20

Summary:

technical-glossary-platform-data-batch-roadmap-v7
technical-glossary-platform-data-batch-v7
technical-glossary-cloud-sre-batch-roadmap-v6
technical-glossary-cloud-sre-batch-v6
technical-glossary-ocp-v5
technical-glossary-cka-v4
technical-glossary-mcp-v3
technical-glossary-rag-v2
technical-glossary-sre-v1

This is important to me. Because one of the biggest risks with personal projects is this: You think you remember what you’ve done. A week later, the details have slipped your mind. That’s why we have a roadmap.

What Have I Gained? This work may seem minor, but it has significantly enhanced Asuli’s reliability. Benefits:

1. Critical technical terms are no longer left to LLM predictions.
2. The risk of incorrect interpretations has been reduced.
3. Asuli provides answers that are better suited to my local context.
4. Regression tests prevent existing behaviour from breaking.
5. Telegram live tests validate the actual usage flow.
6. The roadmap ensures that the rationale behind each change is documented.

The most important lesson, however, is this: I improved the quality of the data input before scaling up the RAG. Because as a poor-quality knowledge base grows, the system does not improve. It simply starts to generate more incorrect answers with greater confidence.

The Next Step: Web Knowledge Ingestion Agent Now the next question is: When I find useful information on the internet, how do I feed it into Asuli? For example: /kimnet napoleon At the moment, Asuli retrieves information from the internet. But this information is fleeting. An answer is provided, and that’s it. The new workflow I’m looking for is as follows:

The name I’ve given it is: Verified Web Knowledge Ingestion Agent But we need to be careful here. Because not all information from the internet is ingested into RAG. The following checks must be carried out first:

1. Is it public information?
2. Does it trigger the private identity guard?
3. Is the source reliable?
4. Is the title a correct match?
5. Is the content sufficient?
6. Does a duplicate record already exist?
7. Is the information time-sensitive?
8. Is the metadata complete?
9. Is it suitable for inclusion in the RAG?

Sample decision: Napoleon → public figure → source: Wikipedia → title match is reasonable → information is permanent → candidate for RAG But: Ali Elmalı → private identity guard → not included in RAG Or: Today’s exchange rate → time-sensitive → cannot be made into permanent RAG information

Suggested Folder Structure Initial design: data/web-cache/ pending/ public_person/ approved/ public_person/ rejected/ public_person/ ingested/ public_person/ Example file: data/web-cache/pending/public_person/napolyon-bonapart.md Metadata:

source_type: web_cache
lookup_type: public_person
query: napolyon
resolved_title: Napolyon Bonapart
source_name: Wikipedia
source_url: https://...
retrieved_at: 2026-06-30
language: tr
quality_status: pending
ingestion_status: not_ingested
risk_level: low
refresh_policy: yearly

Contents:

Napoleon Bonaparte

A brief, well-sourced, clear summary.

Source Note

This entry was taken from the internet and saved to the Asuli web cache. Its inclusion in the RAG requires further approval and quality control.

Why don’t I want to go straight to RAG? Because the RAG knowledge base shouldn’t be a dumping ground. Here’s the rule: A web result can be an answer. But not every web result can be a knowledge base entry. The first version must be controlled:

V1:
- /kimnet results are written to 'pending' as .md files with metadata
- The agent makes a quality/risk decision
- A report is sent via Telegram
- No automatic addition to RAG
V2:
- Approved records are added to the controlled RAG index
- Duplicate checks are carried out
- An ingestion log is maintained
V3:
- Low-risk, clearly matching, permanent public information can be retrieved automatically
This seems more accurate to me.
Because an agentic system is not merely a 'system that performs tasks'.
A good agentic system also does the following:
- checks
- rejects
- provides a justification
- leaves a trail
- reports what it has done and why

Conclusion It may seem as though I haven’t added any major new features to Asuli in this work. But in fact, I have added a fundamental layer of security: A deterministic, tested response that is appropriate to the local context for critical technical terms. For me, this marks an important stage in the maturation of the RAG system.

Because I don’t want to expand the knowledge base without first ensuring the following:

The system should state clearly what it knows.
It should leave what it doesn't know to the normal flow.
It shouldn't drift into the wrong context.
New additions shouldn't disrupt what's already there.

In the next step, I aim to enable Asuli to do the following:

A Web Knowledge Ingestion Agent that evaluates
relevant information from the internet,
stores it using metadata,
rejects it where necessary,
and, where appropriate, feeds it into the local RAG in a controlled manner.

In short: It’s not just about expanding the RAG, but about establishing a secure gateway that decides what enters the RAG. In my view, this is where the real difference in local AI systems begins.

That’s the summary of this section in my journey with the local AI runtime.

**You can read my other articles on Medium.**

I hope you enjoy reading this, and I’d really appreciate it if you could share any relevant experiences you have in the comments.