Semantic Routing: Building block for scalable AI Applications

Large Language Models (LLMs) and Generative AI (GenAI) are changing the way humans interact with technology. These advanced systems are at the core of applications ranging from conversational assistants and automated content creation to sophisticated data analysis tools. The first step behind these applications begins with a simple yet crucial step: understanding user query.

The user’s query serves as the foundation for everything that follows. Whether it’s a request to retrieve information, perform an action, or generate creative output, accurately interpreting the intent behind a query is essential. This initial understanding shapes the system’s ability to provide meaningful, relevant, and precise responses.

Actions taken by AI Application

Why is this step so critical? Because without a clear grasp of the user’s intent, even the most advanced AI systems can falter. Misinterpreting a query can lead to irrelevant information, incorrect actions, or subpar user experiences.

Once the query is understood, AI applications process it in various ways to deliver value. When processing a user query, two components are important to understand to deliver meaningful responses: Intent and Entity. These elements form the backbone of how AI Apps interpret and act upon user requests. Let’s explore these concepts in detail with examples from the wellbeing space:

1. Intent: The “What” Behind the Query

The intent represents the core purpose or action the user wants to achieve through their query. It answers the question: What does the user want the AI system to do?

For example:

• In the query, “Guide me through a 5-minute breathing exercise,” the intent is “start a wellbeing practice.”
• In “What can I do to improve my sleep tonight?” the intent is “seek advice or recommendations.”

2. Entity: The “Who/What/Where/When” of the Query

The entity refers to the specific details or subjects within the query that provide context to the intent. It answers questions like Who? What? Where? When? — essentially the parameters needed to fulfill the intent.

For example:

• In the query, “Guide me through a 5-minute breathing exercise,” the entities are “5-minute” (duration) and “breathing exercise” (activity).
• In “What can I do to improve my sleep tonight?” the entities are “sleep” (goal) and “tonight” (time).

Semantic routing

To understand user queries, extract intent and entities, and direct queries to the appropriate actions, it is essential to set up a dedicated service. This service, referred to as Semantic Routing, acts as building block for large-scale AI applications capable of handling diverse requests and actions. It is a important component needed to build an AI system that can process and respond to queries effectively. Main task for this service will be:

• Understanding User Queries: Analyze and interpret natural language input.
• Extract Intent: Identify the purpose behind the query (e.g., action, information, or content generation).
• Identify Entities: Extract key details like time, activity type, location, or goal.
• Classify and Map: Categorize queries and map them to appropriate actions or services.
• Route Queries: Direct queries to the correct backend systems or modules.
• Handle Diverse Requests: Support various intents and actions seamlessly.
• Incorporate Feedback: Refine routing accuracy using feedback and query outcomes.

How to setup Semantic routing:

Semantic routing can be implemented using various techniques, depending on the complexity of the application and the resources available. Below are different ways to achieve effective semantic routing for intent and entity classification:

1. Open-Source Libraries:

Use NLP libraries like Rasa or spaCy, to build your own classification system. These tools allow you to train models with domain-specific data, enabling precise intent classification and entity recognition. They are ideal when you require flexibility, control, and the ability to fine-tune the system for your specific use case.

2. Using LLM

Large Language Models (LLMs) like GPT-4 or Claude can be leveraged for intent classification and entity extraction using prompt engineering. By crafting precise prompts, LLMs can guide to understand user inputs and generate structured outputs with minimal effort. LLMs are ideal for applications that needs minimal development overhead.

A simple example for intent and entity classification using LLM:

Extract the intent and entities from the following query:
"Schedule a meditation session for tomorrow morning."
Output format:
{
  "intent": "<intent>",
  "entities": {
    "activity": "<activity>",
    "date": "<date>",
    "time": "<time>"
  }
}

{
  "intent": "schedule_activity",
  "entities": {
    "activity": "meditation session",
    "date": "tomorrow",
    "time": "morning"
  }
}

Using LLMs has some downsides. They can be costly, especially for apps with many users, and might be slow, which isn’t ideal for real-time needs. The results depend a lot on how good the prompts are, and they may not work well for specific topics without extra training.

To handle this effectively, I would recommend using LLMs for prototyping due to their ease of setup and ability to deliver quick results. Once the prototype is validated, you can transition to building your own system using an open-source library like Rasa, which offers more control, flexibility, and cost efficiency for long-term use.

3. Prebuilt APIs

Prebuilt APIs like Dialogflow and AWS Lex also offers ready-to-use NLP services for intent and entity classification. These solutions work by processing user input and returning structured outputs and have a pre-trained model. They are best suited for quick setups and projects where advanced customization is not a priority, providing an easy and efficient way to implement basic NLP capabilities.

In addition to above, other approaches like Rule-based systems and Hybrid models are also viable options. Rule-based systems rely on predefined patterns or keyword triggers, making them suitable for narrow domains or small-scale applications. Hybrid systems combine the strengths of multiple methods, such as using LLMs for initial classification and traditional NER for domain-specific entities. These approaches provide flexibility and can be adapted based on the project’s requirements and complexity.

Semantic Routing can be the first filter layer for AI applications, helping to eliminate irrelevant or junk queries before further processing. By accurately understanding user intent and entities, it will ensure that only meaningful requests are routed for appropriate actions, enhancing efficiency, scalability, and overall system performance.