RAG Routers: Semantic Routing with LLMs and Tool Calling

Diagram illustrating a RAG (Retrieval-Augmented Generation) system acting as a semantic router. The system processes the chat history (msg1, msg2, msg3) and retrieves relevant past messages and embeddings from a vector database. It uses tool descriptions to decide which tool (e.g., Summarization, Weather Forecast, Topic Extraction) to invoke, providing the user with the appropriate answer.

Introduction

In the first part of this series (which you can find here), we explored how a Large Language Model (LLM) equipped with various tools could act as a semantic router to handle different user queries effectively. The LLM analyzed the chat history and invoked the appropriate tool based on the user’s request, offering a seamless interaction experience. This approach demonstrated the potential of using LLMs to manage diverse tasks, from summarization to weather forecasting, without the need for extensive pre-training or fine-tuning for each specific task.

Building on this foundation, the second part of this series introduces an enhancement to our system: integrating a vector database to create a Retrieval-Augmented Generation (RAG) system semantic routing. By incorporating a vector database, our LLM now has access to a rich repository of past messages and their corresponding embeddings. This additional context empowers the LLM to make more informed decisions when routing user queries to the appropriate tools.

The vector database serves as a memory bank, allowing the LLM to retrieve relevant historical interactions that may contain similar tool calls or contextual information. This retrieval capability enhances the semantic routing process, improving the accuracy and efficiency of the tool selection. For instance, if a user asks for summarization of recent messages, the LLM can now retrieve past summaries or similar requests, refining its understanding of the task at hand.

In this follow-up article, we will delve into how integrating the vector database enhances our system's semantic routing capabilities. We will explore the architecture, implementation details, and benefits of this approach, highlighting how it transforms the LLM’s ability to handle complex and diverse tasks. By the end of this article, you’ll have a comprehensive understanding of how RAG systems can elevate the performance of LLM-based applications, paving the way for more intelligent and context-aware interactions.

The guide will be developed using Google Colab in Python, utilizing several key packages including PostgreSQL for database management, pgvector for handling vector embeddings within PostgreSQL, and Ollama for instantiating Llama3.2 LLM. This combination of tools provides a robust framework for enhancing the semantic routing capabilities of our system.

The full notebook, including the code and setup details, is available on my **GitHub **repository for reference. In the next section, we will cover the setup process, ensuring you have everything ready to follow along with this tutorial.

Before we dive into the guide, I invite you to **follow my profile** to stay updated on future articles and insights on LLMs, RAG systems, and more. Your support helps bring more innovative solutions to light.

In the following sections, we will walk through the setup process and delve into the details of integrating a vector database with our LLM to enhance its semantic routing capabilities.

Setup

To begin building our semantic router RAG system in Google Colab, we first need to set up PostgreSQL to serve as our vector database. We’ll install PostgreSQL, set up a user and database, and install the pgvector extension for vector storage. Finally, we’ll install the necessary Python packages to interact with the database.

1. Installing and Setting Up PostgreSQL

First, update the package lists and install PostgreSQL along with its contrib package:

!apt update
!apt install postgresql postgresql-contrib

Start the PostgreSQL service:

!service postgresql start

Create a new PostgreSQL user and a database:

!sudo -u postgres psql -c "CREATE USER rag_router WITH PASSWORD 'password';"
!sudo -u postgres psql -c "CREATE DATABASE rag_db OWNER rag_router;"

Check the PostgreSQL version to ensure it’s correctly installed (it should be PostgreSQL 14):

!psql --version

Next, install the necessary server development packages for PostgreSQL:

!sudo apt install postgresql-server-dev-14 #change 14 with the PostgreSQL version downloaded

Clone the pgvector repository, build, and install the extension. For more information visit their GitHub repository.

!git clone --branch v0.8.0 https://github.com/pgvector/pgvector.git
!cd pgvector && make && sudo make install

Restart the PostgreSQL service to apply changes:

!sudo service postgresql restart

Finally, enable the vector extension in the database:

!sudo -u postgres psql -d rag_db -c "CREATE EXTENSION IF NOT EXISTS vector;"

To interact with PostgreSQL from Python, install the following packages:

!pip install sqlalchemy psycopg2-binary pgvector sentence-transformers

With PostgreSQL set up and the necessary packages installed, we’re now ready to proceed with implementing the RAG system. In the next section, we’ll connect to our database and start building the system.

2. Connecting to the Database and Defining the Table

With PostgreSQL and the pgvector extension set up, we can now connect to the database and define the table structure to store our message history and embeddings. We’ll use SQLAlchemy, a popular ORM (Object Relational Mapper) for Python, to handle our database interactions seamlessly.

We start by setting up the connection to the PostgreSQL database using SQLAlchemy. Here’s the code to create the engine and session:

from sqlalchemy import create_engine, Column, Integer, String
from sqlalchemy.orm import sessionmaker, declarative_base

# Define connection details
DATABASE_URL = "postgresql+psycopg2://rag_router:password@localhost:5432/rag_db"

engine = create_engine(DATABASE_URL)
Base = declarative_base()

SessionLocal = sessionmaker(bind=engine)
session = SessionLocal()

This code sets up the connection to our rag_db database using the rag_router user. The SessionLocal is used to create sessions for interacting with the database.

Next, we define the History table to store messages and their corresponding embeddings. We'll use the pgvector extension to store the embeddings as vectors. We specify that the embeddings will be stored as a 384-dimensional vector, as we’ll use a sentence transformer model called all-MiniLM-L12-v2later in our implementation:

from pgvector.sqlalchemy import Vector

class History(Base):
    __tablename__ = "history"

    id = Column(Integer, primary_key=True, index=True, autoincrement=True)
    user_query = Column(String)
    embedding = Column(Vector(384))
    history = Column(String)

Base.metadata.create_all(bind=engine)

Specifically, user_query will store the last message sent by the user, embedding will be the vector representation of the user message, history will contain an example of a conversation with the user and the tool call that was made. In this way the search for emebdding will be done considering only the last message sent by the user, while we will enrich the information provided to the language model by considering the whole example including tool call.

Finally, we create the tables in our database:

Base.metadata.create_all(bind=engine)

With this setup, we’ve established a connection to our PostgreSQL database and defined the schema for storing historical chat messages and their embeddings. This structure will enable the LLM to retrieve past interactions and make informed decisions when routing user queries.

In the next section, we will explore how to generate embeddings using a sentence transformer model and populate our database with historical messages.

Preparing the Dataset and Seeding the Database

Now that we have set up our database, we need to populate it with meaningful data for our RAG system. To do this, we use a sample dataset containing chat interactions, which will allow us to perform similarity searches when retrieving relevant historical messages.

1. Downloading the Dataset

The dataset is a CSV file that contains user messages along with their associated conversation history. I generated this dataset using ChatGPT to simulate a variety of query-response pairs. You can download it using the following command:

!wget https://github.com/Sopralapanca/medium-articles/blob/main/llm-routing/chat_sample.csv?raw=true -O chat_sample.csv

2. Exploring the Dataset

Once the dataset is downloaded, we can load it into a pandas DataFrame and inspect its structure:

import pandas as pd

df = pd.read_csv("chat_sample.csv")
df.head()

Each row in the dataset contains:

• history: A JSON-formatted conversation history that tracks how the query was processed.
• message: The user’s input message.

An example of a row:

{
    "history": "[{\"role\": \"user\", \"content\": \"What is the topic of this statement? 'The construction of the Great Wall of China stood as a symbol of defense and perseverance.'\"}, {\"role\": \"assistant\", \"content\": \"\", \"refusal\": null, \"audio\": null, \"function_call\": null, \"tool_calls\": [{\"id\": \"call_618996\", \"function\": {\"arguments\": \"{}\", \"name\": \"get_topic\"}, \"type\": \"function\", \"index\": 0}]}, {\"role\": \"tool\", \"content\": \"{\\\"result\\\": \\\"The topic of the sentence is The and its significance in history or technology.\\\"}\", \"tool_call_id\": \"call_618996\"}, {\"role\": \"assistant\", \"content\": \"The topic of the sentence is The and its significance in history or technology.\"}]",
    "message": "What is the topic of this statement? 'The construction of the Great Wall of China stood as a symbol of defense and perseverance.'"
}

3. Encoding Messages with Sentence Transformers

To enable efficient similarity search in our vector database, we encode the messages using a sentence transformer model. Here, we use sentence-transformers/all-MiniLM-L12-v2 from Hugging Face, but you are free to use any model that suits your needs.

First, we download and load the model:

from sentence_transformers import SentenceTransformer
model = SentenceTransformer('sentence-transformers/all-MiniLM-L12-v2')

To compute an embedding for a given text, we can use:

# Example string to compute embeddings
example_string = "This is a sample string for embedding."

# Compute the embeddings
embedding = model.encode(example_string)
embedding = embedding.astype(float).tolist()

4. Implementing Similarity Search

With the embeddings stored in our database, we need a method to retrieve the most similar past interactions based on a given query. We implement a similarity search function using L2 distance:

def similarity_search(session, query: str):
    vector = model.encode(query)
    records = (
        session.query(History)
        .order_by(History.embedding.l2_distance(vector))
        .limit(3)
        .all()
    )
    if records:
        return records
    else:
        return None

5. Seeding the Database with Message Embeddings

Now that we have our dataset and encoding model, we can populate the PostgreSQL database with the chat history and message embeddings.

# Add records to PostgreSQL

records = []
for i, row in df.iterrows():
    embedding = model.encode(row["message"]).astype(float).tolist()
    records.append(
        History(
            user_query=row["message"],
            embedding=embedding,
            history=row["history"]
            )
        )

try:
  session.add_all(records)
  session.commit()
except Exception as e:
  print(e)
  session.rollback()
  print("exception raised, rollbacking...")

6. Verifying the Data Insertion

To confirm that the records were inserted successfully, we can query the database:

# Check if data is inserted correctly
history = session.query(History).all()

for msg in history:
    print(f"Message: {msg.user_query}\nEmb: {msg.embedding}\nHistory: {msg.history}\n")
    break

Message: What is the topic of this statement? 'The construction of the Great Wall of China stood as a symbol of defense and perseverance.'
Emb: [ 3.56391110e-02  1.39516190e-01 -5.41455634e-02  1.03664258e-02
  5.39230816e-02  3.67860869e-02  6.48062080e-02 -1.38131268e-02
 -2.76730061e-02 -1.11560058e-02  1.47463987e-02 -5.28952740e-02
  9.83905196e-02 -1.85867175e-02 -1.47388298e-02  1.54635549e-01
 -1.80432014e-02  2.56070755e-02 -4.38161269e-02  1.75370611e-02
  6.85141832e-02 -8.59485939e-02  1.80035885e-02 -1.59655511e-02
  3.60874161e-02 -7.78923854e-02  5.48197068e-02 -2.57355394e-04
  9.83621106e-02  6.14276296e-03 -6.19777888e-02  1.48288729e-02
  2.15548966e-02 -9.35294840e-04  4.90635224e-02  4.82670143e-02
  1.11774608e-01  8.08627345e-03  5.07231131e-02  2.47926861e-02
  4.28537503e-02  4.14998718e-02  6.81590289e-02 -2.15036981e-02
  8.12526420e-02  2.77387793e-03  3.68143395e-02  6.26895428e-02
 -5.55345230e-03 -6.85127266e-03  6.40913546e-02 -3.92634273e-02
 -2.17473134e-02 -5.86708933e-02 -3.40111516e-02  7.87436664e-02
  6.57240022e-03  4.95278835e-02 -4.15265560e-02  9.48650911e-02
 -3.59242558e-02  1.24758534e-01  2.37003900e-02  2.68134382e-02
  3.97864431e-02  9.00040846e-03  3.82129252e-02  7.64965937e-02
  1.03782411e-04  1.88976265e-02 -6.49116933e-02 -1.23188486e-02
 -4.75398637e-03  2.99789850e-02  3.54540278e-03  5.00601949e-03
  8.93896259e-03  3.41104530e-02  4.67196945e-03 -1.00443669e-01
  3.16508338e-02  1.45623286e-03 -4.84539643e-02  7.12452531e-02
 -9.68752429e-03 -9.10218284e-02 -5.55495620e-02 -3.10241226e-02
 -4.22470417e-04 -1.19265001e-02 -3.66672128e-02  1.46445353e-02
  4.96426970e-02  6.59978241e-02  1.02954879e-01 -3.09354644e-02
 -3.20205912e-02 -5.73691204e-02  3.43689173e-02  6.34342879e-02
  2.55851224e-02  6.27490729e-02  1.06596723e-02 -8.67631882e-02
 -6.63076481e-03  4.63875290e-03 -3.32081765e-02 -5.73190972e-02
 -3.84991691e-02 -2.44544484e-02  8.25658254e-03 -2.55440194e-02
  1.12267323e-02  7.35786632e-02  2.79526003e-02 -4.73383553e-02
 -1.58503968e-02  1.05204647e-02 -1.86165944e-02 -2.96575539e-02
  7.48385042e-02 -2.12699007e-02  5.57069527e-03 -1.62029061e-02
 -6.61655515e-02 -9.45105404e-02  4.32871394e-02 -4.66416441e-02
  2.62509473e-02  4.39372985e-03 -1.38002979e-02  1.33248335e-02
 -8.18160269e-03 -8.02748129e-02  2.70134453e-02  1.23223849e-02
 -7.47878617e-03 -3.02926470e-02 -4.14032638e-02 -5.72613478e-02
  3.09377201e-02  1.01244375e-01  3.20321359e-02 -2.97183786e-02
 -1.10303320e-01 -2.09869090e-02 -4.43669371e-02 -1.02053648e-02
  8.20063427e-02  7.62664452e-02 -8.12128708e-02 -5.02047921e-03
  5.78943007e-02  1.44760739e-02  4.03009541e-02  7.63541367e-03
 -8.50614384e-02  1.66245596e-03  8.43952596e-03 -2.15932019e-02
 -4.56671417e-02  4.75338548e-02 -3.62151600e-02  4.73676398e-02
 -2.97146812e-02 -1.55824600e-02 -8.67354870e-03  2.77717058e-02
  1.74703321e-03  1.81635134e-02  2.87144631e-03 -5.41020604e-03
  1.12942189e-01  4.22507748e-02  3.91682814e-04 -8.53203014e-02
 -4.47661914e-02 -4.76459078e-02 -3.29743652e-03 -1.56637095e-02
 -4.58651893e-02 -4.28264774e-02  2.43080705e-02 -5.89831397e-02
 -9.11172256e-02  3.48661244e-02  8.31293017e-02  4.01404984e-02
 -3.10640279e-02  1.24621168e-02 -2.74060816e-02  1.26842916e-01
 -2.19727959e-02  9.75351129e-03 -5.09146526e-02 -1.67063437e-02
 -3.23992246e-03  2.87737064e-02 -1.79229793e-03  3.00386106e-04
  3.36830020e-02 -1.28349945e-01 -5.14890850e-02 -3.70209031e-02
 -1.97049398e-02 -1.65791754e-02  3.59189548e-02 -3.81764360e-02
 -7.11437687e-02 -3.98273207e-02  8.34823996e-02  4.33139391e-02
 -1.99118759e-02 -2.18496043e-02  8.09674039e-02 -6.74301088e-02
  5.46612293e-02 -4.60898355e-02  2.00929865e-02 -4.15070616e-02
  6.55146763e-02  1.45574342e-02 -9.53789726e-02  1.94731380e-32
 -4.41746990e-04  8.25959083e-04  2.97364648e-02 -7.46690028e-04
  7.88325816e-02 -4.50695232e-02 -6.51104674e-02 -8.05454850e-02
 -7.88509473e-03  3.96083333e-02 -4.75353599e-02 -2.30434928e-02
 -6.51504025e-02  2.76793819e-02 -5.01169302e-02 -6.53670430e-02
 -5.98709285e-02  4.23941482e-03 -8.36451054e-02  1.13330772e-02
 -2.34593190e-02 -4.59063910e-02  3.57888849e-03 -4.64538410e-02
 -2.98117474e-02  7.94593543e-02 -2.64531281e-02 -1.59601942e-01
  4.22903225e-02 -2.06399094e-02  2.75755525e-02  1.84438974e-02
 -9.15093422e-02  5.66963106e-02  1.66896638e-02 -3.73455472e-02
  1.52522415e-01 -8.48315880e-02  2.63659228e-02 -3.79037857e-02
  1.38341263e-02  4.33253236e-02 -2.34484095e-02  9.58907083e-02
 -1.31546184e-01 -1.39121357e-02  4.28867005e-02 -2.43927296e-02
  4.78442991e-03 -1.73287131e-02 -6.86097741e-02  3.76354791e-02
  1.17679916e-01 -4.43245620e-02  3.07631288e-02  1.20571703e-02
 -2.17988212e-02  6.58176616e-02  1.77962612e-02 -4.60228510e-02
  7.70573970e-03  3.27375829e-02  6.07456602e-02 -4.94542122e-02
  1.00167699e-01 -2.38578729e-02  4.65676300e-02  4.62845601e-02
 -4.97905863e-03  1.85510435e-03 -1.16905391e-01  3.51430066e-02
 -1.06304400e-01  8.91971216e-02 -3.10810525e-02  5.48285544e-02
 -3.63852121e-02 -2.21131593e-02  3.69688943e-02  1.48576617e-01
  2.97230985e-02 -6.04069978e-02  1.77705847e-02 -5.33979535e-02
  2.62360368e-02 -6.37743762e-03 -7.56596848e-02  2.04473902e-02
  7.99300298e-02  2.35495856e-03 -3.24344523e-02 -1.81301851e-02
 -1.31327033e-01 -3.24439183e-02 -7.39989849e-03  4.80660492e-32
  2.95547321e-02 -7.94379134e-03 -9.85979363e-02  3.08770710e-03
  1.08095780e-02  2.17941068e-02  9.08277463e-03 -3.99402045e-02
  1.82272941e-02  3.61790732e-02  2.63478216e-02  1.13106752e-02
 -6.57167435e-02 -3.85014787e-02 -4.57616001e-02 -1.21498890e-02
  1.38666248e-02 -7.27558360e-02  1.94157399e-02 -2.25873925e-02
  4.49696966e-02 -2.24020313e-02 -5.85773066e-02 -5.40203154e-02
 -3.80695499e-02  6.43591136e-02  1.37437014e-02  5.77405235e-03
  1.27854757e-02 -2.48799287e-02  1.23569416e-03 -1.18192136e-02
 -2.60241125e-02 -5.27605973e-02  4.36215987e-03 -9.67641175e-03
  4.66862544e-02 -5.00502773e-02  7.56993815e-02 -7.21424446e-02
 -3.45649831e-02  6.74557760e-02  2.10934952e-02  3.93601432e-02
 -1.10357683e-02  3.09072342e-02  9.55893192e-03  3.02247368e-02
 -2.37568989e-02 -4.54251878e-02 -7.33901784e-02  2.13674009e-02
  7.13499486e-02 -2.92651337e-02 -7.71111762e-03  1.10905610e-01
  2.95666140e-02  4.70999219e-02  9.64558974e-04 -4.27337065e-02
  3.81121859e-02  5.68572134e-02 -1.32158529e-02  5.79213873e-02]
History: "[{\"role\": \"user\", \"content\": \"What is the topic of this statement? 'The construction of the Great Wall of China stood as a symbol of defense and perseverance.'\"}, {\"role\": \"assistant\", \"content\": \"\", \"refusal\": null, \"audio\": null, \"function_call\": null, \"tool_calls\": [{\"id\": \"call_618996\", \"function\": {\"arguments\": \"{}\", \"name\": \"get_topic\"}, \"type\": \"function\", \"index\": 0}]}, {\"role\": \"tool\", \"content\": \"{\\\"result\\\": \\\"The topic of the sentence is The and its significance in history or technology.\\\"}\", \"tool_call_id\": \"call_618996\"}, {\"role\": \"assistant\", \"content\": \"The topic of the sentence is The and its significance in history or technology.\"}]"

Now that our database is seeded with sample chat messages and embeddings, we are ready to integrate our LLM-powered retrieval system with PostgreSQL. In the next section, we will download Ollama and Llama3.2. Next, we will use the vector database to improve our LLM routing mechanism.

Setting Up Ollama and Large Language Models (LLMs)

What is Ollama?

Ollama is a powerful framework designed to run and manage large language models (LLMs) efficiently on local machines. It allows users to download, serve, and interact with models without needing external API dependencies. With Ollama, developers can fine-tune model deployment and optimize execution for various tasks, from chatbot applications to complex AI-driven workflows.

Downloading and Setting Up Ollama

To install Ollama, run the following command:

!curl https://ollama.ai/install.sh | sh

Once installed, start the Ollama server:

!ollama serve > server.log 2>&1 &  # Ollama runs by default at 127.0.0.1:11434

Downloading Large Language Models

For this implementation, we use llama3.2:3b both as a router and as specialized task agents. This choice is made for simplicity and illustration purposes. Users can select different models for routing and specialized tasks as long as the chosen routing model supports tool calls.

To download the models, execute:

%%capture
agents_model = "llama3.2:3b"
#!ollama pull {agents_model}

%%capture
router_model = "llama3.2:3b"
!ollama pull {router_model}

Check the available models:

!ollama ls

NAME           ID              SIZE      MODIFIED               
llama3.2:3b    a80c4f17acd5    2.0 GB    Less than a second ago 

Serving Models at Different Ports

To serve multiple models simultaneously on different ports, set environment variables and run multiple instances of Ollama:

import os

os.environ['OLLAMA_HOST'] = "127.0.0.1:11438"
!ollama serve > server.log 2>&1 &

os.environ['OLLAMA_HOST'] = "127.0.0.1:11439"
!ollama serve > server.log 2>&1 &

Setting Up Router Model and Specialized Agents

Before defining the router model and its tools, install the necessary Python packages:

!pip install openai==1.57.4 ollama==0.4.4 pydantic==2.10.3

These packages allow us to interact with Ollama, define structured data using Pydantic, and use OpenAI’s API.

Defining Available Tools

In a function-calling LLM setup, tools act as specialized functions the model can invoke when a user requests specific tasks. We define two tools:

from openai import OpenAI
import openai

summarizer_tool = {
    "type": "function",
    "function": {
        "name": "get_summarization",
        "description": "Call this method whenever you need to perform a summarization task, for example when a user asks 'Summarize this paragraph'. This method does not accept input parameters.",
        "parameters": {
            "type": "object",
            "properties": {},
            "required": [],
            "additionalProperties": False
        }
    }
}

topics_tool = {
    "type": "function",
    "function": {
        "name": "get_topics",
        "description": "Call this whenever you need to perform a topic extraction task, for example when a user asks 'What are the topics of this paragraph?'. This method does not accept input parameters.",
        "parameters": {
            "type": "object",
            "properties": {},
            "required": [],
            "additionalProperties": False
        }
    }
}
tools = [topics_tool, summarizer_tool]

These tools are designed for two tasks:

• Summarization: Extracts a concise summary of a given text.
• Topic Extraction: Identifies key topics within a paragraph.

Please note that in this case we do not have the language model generate the input parameters for the tool. This is because, for the chosen tasks, the LLM might invent or change the input text. Instead we will pass to the tool exactly the user’s last message

For simplicity, we define only two tools, but additional tools can be created by following the same structure. The descriptions must be precise, as ambiguous descriptions may confuse the model when deciding which tool to call.

Defining Specialized Agents

Each specialized agent is responsible for handling a specific task. The summarizer and topic extractor agents are defined as follows:

from openai import OpenAI

summarizer_client = OpenAI(
    base_url='http://localhost:11439/v1',
    api_key='ollama',  # required, but unused
)

topic_client = OpenAI(
    base_url='http://localhost:11438/v1',
    api_key='ollama',  # required, but unused
)

• The summarizer_client connects to the model running on port 11439, which handles summarization.
• The topic_client connects to the model running on port 11438, which handles topic extraction.

We then define a helper function to construct conversation history:

def build_history_agents(messages: list[dict[str, str]], sys_prompt: str):
    history = [{"role": "system", "content": sys_prompt}]
    for message in messages:
        if message["role"] != "system":
            history.append(message)
    return history

This function ensures that:

• Each agent receives a system message with clear role instructions.
• The conversation history is maintained correctly.

Summarization Agent

def get_summarization(history: list[dict[str, str]]):
    print("calling summarization model")

    sys_prompt = "You are a summarizer agent. Your only role is to summarize user messages. If the user requests any other action or chit-chat, just answer 'Pass.'"
    k = build_history_agents(history, sys_prompt)

    response = summarizer_client.chat.completions.create(
        model=agents_model,
        messages=k,
    )

    return response.choices[0].message.content

This function:

• Calls the summarization model.
• Enforces a strict role for the agent (it only summarizes and ignores unrelated requests).
• Returns the generated summary.

Topic Extraction Agent

def get_topics(history: list[dict[str, str]]):
    print("calling topic model")

    sys_prompt = "You are a topic extractor model. Your role is to examine user messages and extract the topics. If the user requests any other action or chit-chat, just answer 'Pass.'"

    response = topic_client.chat.completions.create(
        model=agents_model,
        messages=build_history_agents(history, sys_prompt),
    )

    return response.choices[0].message.content

This function:

• Calls the topic extraction model.
• Uses a strict prompt to ensure the agent does not perform other tasks.
• Returns the extracted topics.

Utility Functions

Executing Function Calls

import json
import re

def execute_function_call(response, history):
  if response.choices[0].finish_reason == "tool_calls":
    tool_call = response.choices[0].message.tool_calls[0]
    function_name = tool_call.function.name

    result = globals()[function_name](history)

    function_call_result_message = {
      "role": "tool",
      "content": json.dumps({
          "result": result
      }),
      "tool_call_id": response.choices[0].message.tool_calls[0].id
    }

    response_dict = response.model_dump()
    response_dict["choices"][0]["message"]

    history = history + [response_dict["choices"][0]["message"]] + [function_call_result_message]

    return history

This function manages the execution of tool calls. If the model calls a tool, it retrieves the function name and executes it. The tool’s response is formatted and appended to the conversation history.

Building Router History

def build_router_history(messages: list[dict[str, str]], response):
    res = response.choices[0].message.content
    messages.append(
        {
            "role": "assistant",
            "content": res
        }
    )
    print(res)
    return messages

This, appends the router model’s response to the message history and helps in maintaining context across multiple turns.

Router

The router model determines which tool or agent should handle a user request.

router_client = OpenAI(
    base_url='http://localhost:11434/v1',
    api_key='ollama',  # required, but unused
)

The function that processes requests using the router:

import copy

router_client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama', # required, but unused
)

def call_router(messages: list[dict[str, str]], use_tools=True):

  user_request = messages[-1]["content"]

  # retrieve similar query
  res = similarity_search(session, user_request)

  augmented_history = copy.deepcopy(messages)
  augmented_message = f"User request: {user_request}\n"

  if res:
    results = "Retrieved results:\n"
    for record in res:
        results += record.history + "\n\n"

    augmented_message += results

  augmented_history[-1]["content"] = augmented_message

  response = router_client.chat.completions.create(
    model=router_model,
    messages=augmented_history,
    tools=tools if use_tools else None
  )

  return response

This function:

1. Processes the user’s request and augments it with retrieved information (if available).
2. Passes the request to the router model, which decides whether to:

• Handle the request directly.
• Invoke a specialized agent via function calling.

1. Uses function calling if enabled (use_tools=True) to allow routing to the appropriate tool

Chatting with the Assistant

Now that we have set up our router model and specialized agents, we can interact with the assistant. The router’s job is to analyze the chat history and user intent, determine which tool to use, and invoke the appropriate specialized model. If a request doesn’t require a specialized tool, the router will answer directly.

Below, we define a system prompt that guides the router on how to handle user requests.

System Prompt Explanation

The system prompt is crucial because it defines how the router should behave. It informs the model that it acts as a RAG routing model, meaning it should analyze both the user request and retrieved historical data to decide which tool to call.

Key elements of the system prompt:

1. Defining the Router’s Role: The router understands user intent based on chat history and available tools.
2. Listing Available Tools: get_summarization used to summarize a paragraph,get_topics used to extract topics from a paragraph.
3. Guidelines for Answering: if the request matches a tool, call the tool. If the request does not require a tool, respond directly.
4. How User Messages Are Structured: The user request is formatted as:

• User request: -> actual user message
• Retrieved results: -> list of similar requests retrieved from a database

Code: Interacting with the Assistant

We create a conversation history and send a summarization request.

sys_prompt = """
You are a RAG routing model. Your role is to understand the chat history and user intent and use available tools to answer the questions.
Your available tools are:
- get_summarization: use it to summarize a user paragraph.
- get_topics: use it to extract the topics from a user paragraph.

For all other requests you can answer directly.

User messages will be provided to you in the form of:
'User request: -> actual user message'
'Retrieved results: -> list of similar requests retrieved from a database that you can use to understand which tool to call'
"""

messages = [
  {
      "role": "system",
      "content": sys_prompt
  },
  {
      "role": "user",
      "content": "Hi, can you summarize this paragraph? The red glow of tail lights indicating another long drive home from work after an even longer 24-hour shift at the hospital. The shift hadn’t been horrible but the constant stream of patients entering the ER meant there was no downtime. She had some of the “regulars” in tonight with new ailments they were sure were going to kill them. It’s amazing what a couple of Tylenol and a physical exam from the doctor did to eliminate their pain, nausea, headache, or whatever other mild symptoms they had. Sometimes she wondered if all they really needed was some interaction with others and a bit of the individual attention they received from the nurses."
  }
]

response = call_router(messages)
print(response)

We obtain as response:

ChatCompletion(id='chatcmpl-793', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_i71ihvtb', function=Function(arguments='{}', name='get_summarization'), type='function', index=0)]))], created=1742753707, model='llama3.2:3b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=15, prompt_tokens=1222, total_tokens=1237, completion_tokens_details=None, prompt_tokens_details=None))

When analyzing the response, we observe that the LLM correctly calls the get_summarization tool. We now execute the tool call.

fun_call_result = execute_function_call(response, messages)

Analyzign the result of the function call we obtain:

calling summarization model
[{'role': 'system',
  'content': "\nYou are a RAG routing model. Your role is to understand the chat history and user intent and use available tools to answer the questions.\nYour available tools are:\n- get_summarization: use it to summarize a user paragraph.\n- get_topics: use it to extract the topics from a user paragraph.\n\nFor all other requests you can answer directly.\n\nUser messages will be provided to you in the form of:\n'User request: -> actual user message'\n'Retrieved results: -> list of similar requests retrieved from a database that you can use to understand which tool to call\n"},
 {'role': 'user',
  'content': 'Hi, can you summarize this paragraph? The red glow of tail lights indicating another long drive home from work after an even longer 24-hour shift at the hospital. The shift hadn’t been horrible but the constant stream of patients entering the ER meant there was no downtime. She had some of the “regulars” in tonight with new ailments they were sure were going to kill them. It’s amazing what a couple of Tylenol and a physical exam from the doctor did to eliminate their pain, nausea, headache, or whatever other mild symptoms they had. Sometimes she wondered if all they really needed was some interaction with others and a bit of the individual attention they received from the nurses.'},
 {'content': '',
  'refusal': None,
  'role': 'assistant',
  'audio': None,
  'function_call': None,
  'tool_calls': [{'id': 'call_i71ihvtb',
    'function': {'arguments': '{}', 'name': 'get_summarization'},
    'type': 'function',
    'index': 0}]},
 {'role': 'tool',
  'content': '{"result": "A nurse worked two shifts in the hospital due to constant patient flow, leaving her little time for rest or downtime. She dealt with \\"regulars\\" who seemed severe, but were often treated with relatively simple medications, leading her to wonder if some patients simply needed human interaction and care instead of urgent treatment."}',
  'tool_call_id': 'call_i71ihvtb'}]

As you can see from the last message, actually we have a summarization of the paragraph computed by the summarization agent. So pass it back to the router.

response = call_router(fun_call_result, use_tools=False)
res = build_router_history(messages, response)

Which will answer:

The nurse worked two long shifts in the hospital due to constant patient flow, leaving her little time for rest or downtime. She dealt with "regulars" who seemed severe, but were often treated with relatively simple medications, leading her to wonder if some patients simply needed human interaction and care instead of urgent treatment.

Now, we ask the assistant to extract topics from the first message.

messages.append({
      "role": "user",
      "content": "Now can you extract topics from the paragraph that I sent you on the first message?"
  }
)

response = call_router(messages)
print(response)

Again, we analyze the response and see that the router correctly calls the get_topics tool.

ChatCompletion(id='chatcmpl-648', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', refusal=None, role='assistant', audio=None, function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='call_ipcwovno', function=Function(arguments='{"$":"[]"}', name='get_topics'), type='function', index=0)]))], created=1742753722, model='llama3.2:3b', object='chat.completion', service_tier=None, system_fingerprint='fp_ollama', usage=CompletionUsage(completion_tokens=17, prompt_tokens=1155, total_tokens=1172, completion_tokens_details=None, prompt_tokens_details=None))

We then execute the function call.

fun_call_result = execute_function_call(response, messages)
response = call_router(fun_call_result, use_tools=False)
res = build_router_history(messages, response)

At this point, the topics are correctly extracted.

Here are the topics extracted from the original paragraph:

1. Nursing
2. Hospital shift
3. Patient care
4. ER (Emergency Room) environment
5. Human interaction
6. Physical and mental well-being of patients

Conclusion

In this section, we demonstrated how a retrieval-augmented generation (RAG) system can serve as a semantic router, effectively directing user queries to the appropriate specialized models. By defining a clear system prompt and specifying available tools, we enabled the router to make informed decisions on whether to respond directly or invoke a tool.

We tested this by:

• Requesting a summary of a paragraph, which correctly triggered the get_summarization tool.
• Asking for topics from the same paragraph, which prompted the system to call the get_topics tool.

Impact of the System Prompt and Tool Definitions

The accuracy of the router’s decisions depends significantly on how the system prompt is written and how the tools are defined. A well-structured prompt ensures that the model understands when to call a tool and which tool to choose. The more clearly each tool is described, the less ambiguity there is, reducing the risk of incorrect function calls.

By modifying the system prompt or adding more tools, we can extend the router’s capabilities to handle additional tasks such as text classification, sentiment analysis, or translation. This flexibility makes the system highly adaptable to different applications.

Using Different Models for Different Tasks

One of the strengths of this approach is its modular design, which allows different models to handle different tasks. The router itself can be implemented using a general-purpose language model, while specialized tasks like summarization or topic extraction can be handled by models optimized for those specific domains. This flexibility allows for better trade-offs between accuracy, efficiency, and cost.

Advantages of Using RAG as a Semantic Router

A RAG-based router provides several key benefits over traditional rule-based approaches:

• Better intent understanding: The system can infer user intent by analyzing both the chat history and retrieved examples.
• Context-aware decision-making: Retrieved past interactions help refine tool selection and improve response quality.
• Dynamic adaptability: The system can evolve by integrating new tools and models as requirements change.

By leveraging retrieval-augmented generation (RAG) with a structured routing mechanism, we enhance the efficiency, accuracy, and flexibility of AI-driven assistants. This approach allows language models to move beyond generic responses and provide task-specific, context-aware interactions, making them more reliable and effective for real-world applications.

If you liked the article and want to help me write better and better articles, don’t forget to follow me and contribute a tip by clicking this link or the buttons below :) Thank you very much!