Unlocking RAG’s Potential: The Power of Text Embeddings

Hi guys, first of all thanks for your continued support! Medium released a post indicating major policy modifications focused on AI-generated content because writers who do not use AI face harm when their work receives equal treatment. The reason I need to share this information with you is important. The entire content I create goes through AI to ensure it meets a certain standard and is often toned down or up-scaled depending on how the article proceeds.

Since I can no longer do that, you will notice some considerable changes in the language, overall structure and vocabulary. I often try to challenge myself when taking down notes, as the best way to tortu..I mean improve yourself is to do it now. So please try to adjust I shall try my best to keep my articles concise. Thanks for reading this to the end!

If you are new to the concept of RAG, you should check out my previous article which provides a superficial view on RAG. Link’s below:

[embed]Augmenting Intelligence: Unlocking the Power of RAG in NLP Retrieval-Augmented Generation (RAG) is an advanced technique in Natural Language Processing (NLP) that combines…medium.com

Unveiling The Role of Embeddings in RAG Systems

A well-built Retrieval-Augmented Generation (RAG) system requires embeddings as its foundation to connect unstructured data to structured queries through semantic processing. This study investigates the diverse functions of embeddings within RAG systems, along with their performance effects and implementation strategies. The process of embedding natural language into a mathematical form enables RAG systems to retrieve precise information, which benefits numerous application domains.

The ability of RAG systems to retrieve relevant information depends primarily on the embedding. When RAG applies embeddings to raw text, it produces high-dimensional vector spaces that preserve semantic relationships through vectors that remain proximate to one another. The conversion process produces an organized structure which enables quick matching between user requests and available knowledge bases.

Embeddings enable semantic relationships in RAG, visualized as high-dimensional vector clusters.

The RAG pipeline leverages embeddings at several stages. Document embeddings are generated in the preprocessing phase using embedding models which transform content into vector representations after chunking the material. The system converts user queries into query embeddings which are comparable to document embeddings stored inside vector databases.

Embedding Mechanisms: Detecting Semantic Relationships and Contextual Language Meanings

Embeddings operate through an approach that detects semantic relationships between words using mathematical equations. Current embedding models implement transformer architectures to generate dense numerical representations which capture contextual language meanings. The embedding space contains different semantic features of text content and arranges analogous concepts.

Transformer embeddings capture semantic relationships, powering Faiss search

The embedding procedure starts by tokenizing the text, followed by neural network processing and vector representation extraction from model layers or positions. The vectors serve as foundational units to determine the similarity measurements between them, whereas cosine similarity and dot product metrics are the primary computational methods. (A proper discussion regarding cosine similarity and dot product metrics is yet to come, for now I just want you to be introduced to these terms) .The dimensions in embeddings differ between models, starting from 384 dimensions in all-MiniLM-L6-v2 up to 1,536 dimensions which text-embeddings-ada-002 provides from OpenAI.

Tokens to vectors: neural networks encode text into high-dimensional embeddings for cosine similarity.

Generating Embeddings for RAG

The embeddings used in RAG applications are mainly dense and sparse. Deep learners, such as OpenAI and Sentence Transformers, create dense embeddings which pack text data into a condensed vector structure that includes significant values throughout its elements.

Hybrid RAG embeddings: uniting dense semantics with sparse lexical precision for advanced retrieval.

The generation of sparse embeddings utilizes TF-IDF or BM25 methods which produce vectors with high dimensionality, while most elements remain at zero value. SPLADE (Sparse Lexical and Expansion Model) applies the BERT architecture to advanced sparse models which generate information-rich vectors that unite keyword exactness with semantic meaning. Hybrid embedding structures made of dense and sparse approaches have becomes standard in advanced RAG systems for merging semantic understanding with lexical precision.

Unsure about BERT, Word2Vec and other models? They have been discussed in detail in a separate article on Embeddings models, if you are curious feel free to go through this:

[embed]Embedding Models: A Comprehensive Guide for Beginners to Experts Revolutionize your NLP skills: Master word embeddings, contextualized models, and cutting-edge techniques to unlock…medium.com

Embedding Model Selection in RAG Systems

The selection of the embedding model directly shapes how the RAG performs its operations. Models receive assessment through normalized discounted cumulative gain (NDCG) evaluation to determine their ability to place relevant documents at higher positions within the retrieved results. Several essential factors should be considered during the selection process of embedding models:

• The main performance indicator for model accuracy in identifying relevant information is called the retrieval performance.
• The dimensions of the model determine both the required computational power and processing speed.
• The maximum length of tokens defines the amount of text which an embedding process can handle at once.
• The model’s ability to handle domain-specific vocabulary and subject matter represents domain alignment.

I know you tried to scroll horizontally ;)

Model size is directly related to retrieval performance. However, this relation creates and issue for both computational efficiency and necessary latency. The relationship between model size and resource availability depends on particular business applications and existing organizational assets.

Optimizing Retrieval with Vector Databases and ANN Algorithms

The performance and scalability of the entire RAG infrastructure depend heavily on the embedding component. The quality of Embeddings supports better storage and retrieval operations, which decreases the number of documents processed by the generative model. System efficiency directly results in shorter response times and reduced computational expenses.

The vector database Milvus operates as an embedded data storage solution that implements optimized retrieval algorithms for quick search operations. Approximate Nearest Neighbour (ANN) algorithms serve as vector databases to enhance fast similarity lookup functions across extensive datasets, but performance speed might sacrifice some precision accuracy. Another important strategic decision regarding RAG performance involves selecting vector databases together with appropriate indexing methods.

Domain-Specific Embeddings: Fine-Tuning for Precision and Efficiency

General-purpose embedding models do not embrace the specialized knowledge essential for sociotechnical (I read a new word I used it) environments. When embedding models receive domain-specific training data, they become effective at retrieving information for specialized RAG systems. Embeddings model receive training at the target domain through specific datasets to learn domain-specific terminology with a concurrent understanding of conceptual relationships.

Through the Sentence Transformers library, developers access tools that help model fine-tuning using contrastive learning methods and several alternative approaches. Matryoshka Representation Learning (MRL) allows users to create embeddings with truncateable dimensions that preserve 99.9% performance while requiring three times less storage space because the system puts crucial details first in the embedding dimensions. This benefits applications that have storage capacity restrictions.

Efficient MRL: Truncateable Sentence Transformer embeddings preserve 99.9% performance with 3x reduced storage

The article was supposed to have a code example but our RAG knowledge has grown extensive so I decided to challenge myself with a slightly advanced project. The next article will bring you a basic project which focuses on website retrieval from RAG through API key implementation. This provides a convenient way to update your appearance and lets you post the results on social media platforms to show your progress with RAG.

Glossary

• Embeddings: Functions as mathematical text representations which form the base for RAG systems to execute accurate information retrieval.
• Vector Space: Creates text representations through vectors that place related semantic content near one another in high-dimensional mathematical spaces.
• Semantic Relationships: Embeddings use mathematical equations to detect meaningful word relationships known as semantic relationships.
• Cosine Similarity: A primary computational method used to measure similarity between vector embeddings in RAG systems.
• Dense Embeddings: Compressed vector structures which OpenAI and Sentence Transformers produce through their models while maintaining vital information across their vector components.
• Sparse Embeddings: High-dimensionality vectors that use TF-IDF or BM25 algorithms keeping most values at zero to prioritize exact keyword matches.
• SPLADE: Spare Lexical and Expansion Model that applies BERT architecture to create information-rich vectors combining keyword exactness with semantic meaning.
• Hybrid Embeddings: Structures combining dense and sparse approaches to merge semantic understanding with lexical precision.
• Tokenization: The initial step in the embedding procedure where text is broken into processable units.
• Vector Representation: The numerical form of text extracted from model layers or positions after neural network processing.
• Document Embeddings: Vector representations of content generated during the preprocessing phase of the RAG pipeline.
• Query Embeddings: Vector representations of user requests that can be compared to stored document embeddings.
• Vector Database: Storage solutions like Milvus that implement optimized retrieval algorithms for quick search operations.
• Approximate Nearest Neighbour (ANN) : Algorithms that enhance fast similarity lookup across extensive datasets, sometimes sacrificing precision for speed.
• Normalized Discounted Cumulative Gain (NDCG): Evaluation metric used to assess embedding models’ ability to place relevant documents higher in retrieved results.

Citations:

1. https://www.mongodb.com/developer/products/atlas/choose-embedding-model-rag/
2. https://www.philschmid.de/fine-tune-embedding-model-for-rag
3. https://zilliz.com/learn/beginner-guide-to-website-chunking-and-embedding-for-your-genai-applications
4. https://unstructured.io/blog/understanding-embedding-models-make-an-informed-choice-for-your-rag
5. https://www.matillion.com/blog/a-deep-dive-into-embedding-and-retrieval-augmented-generation-rag
6. https://www.mongodb.com/developer/products/atlas/gemma-mongodb-huggingface-rag/
7. https://python.langchain.com/docs/tutorials/rag/
8. https://wandb.ai/mostafaibrahim17/ml-articles/reports/Vector-Embeddings-in-RAG-Applications--Vmlldzo3OTk1NDA5
9. https://pixion.co/blog/designing-rag-application-a-case-study
10. https://community.openai.com/t/rag-and-embeddings-is-it-better-to-embed-text-with-labels-or-not/604100
11. https://docs.mistral.ai/guides/rag/
12. https://arxiv.org/abs/2407.09252
13. https://www.samsungsds.com/en/insights/rag-customization.html
14. https://community.openai.com/t/rag-is-failing-when-the-number-of-documents-increase/578498