Building a Grounded RAG Agent on Azure AI Search and Azure AI Foundry

Introduction

Picture a digital forensics training provider that has years of course material sitting in PDFs: lecture slides, lab guides, reference chapters. Students and new analysts constantly need to find one specific concept buried somewhere in dozens of decks, and searching by keyword across scattered files is slow and unreliable. What they actually want is to ask a question in plain language and get an answer pulled directly from the course material itself, not a generic textbook summary that might not match what was actually taught.

That’s the exact shape of Retrieval-Augmented Generation (RAG): a language model that doesn’t answer from its own training data, but retrieves relevant chunks from a specific document set and grounds its response in them. This build walks through standing up that pattern end to end in Azure: Azure AI Search indexing a set of real PDFs with vector embeddings, and an Azure AI Foundry agent using that index to answer questions grounded in the source material rather than general knowledge.

It also turned into one of the more instructive debugging arcs of this series so far. Almost nothing here failed because the concept was wrong; it failed because of small, specific mismatches between resources, auth models, and default agent behavior. Each of those is documented below rather than smoothed over, since the friction points are genuinely more useful to a reader than a clean walkthrough would be.

Understanding the Architecture

Four Azure services do four distinct jobs here.

Blob Storage holds the source documents. In this build, that’s a set of digital forensics course PDFs, but the pattern is document-agnostic; any private, sensitive, or domain-specific document set fits the same shape.

Azure AI Search indexes and retrieves. This is the retrieval half of RAG. A skillset chunks each document into smaller pieces, calls an embedding model to convert each chunk into a vector, and stores both the text and the vector in a searchable index. At query time, AI Search finds the chunks most semantically similar to the question, not just keyword matches.

Azure AI Foundry hosts the models and the agent. Two model deployments live here: an embedding model (used at indexing time and query time to vectorize text) and a chat model (used to generate the actual answer). The agent itself is a Foundry construct that wraps the chat model together with a knowledge source and a set of instructions, turning “a model” into “a configured assistant with a defined job.”

Managed Identity and RBAC gate who can query what. Every connection between these services (Search to Blob Storage, Search to the embedding model, the agent to the Search index) is intended to run on scoped roles rather than embedded keys wherever possible. This build didn’t get all the way to a fully key-free chain, and that gap is documented honestly in the sections below rather than glossed over.

Stage One: Source Documents

A private blob container (knowledge-docs) was created and populated with a set of real course PDFs. Access was kept private, no anonymous read, consistent with treating the source material as sensitive content rather than public data.

Stage Two: Model Deployments in Azure AI Foundry

Two models were deployed inside a Foundry project:

• A chat model (chat-deploy), used to generate answers
• An embedding model (embed-deploy, text-embedding-3-small), used to vectorize both the source documents and incoming queries so they can be compared in the same vector space

Why two separate deployments, and why these two roles specifically: a RAG pipeline needs two fundamentally different capabilities, and a single model can’t efficiently do both. Retrieval needs a way to turn text into a numeric representation (a vector) that captures meaning, so that “semantic similarity” can be computed as a distance calculation rather than a keyword match. That’s an embedding model’s entire job, and nothing else; it doesn’t generate prose, it just maps text to vectors. Generation needs the opposite: a model that can read retrieved context and produce a fluent, coherent answer in natural language. That’s the chat model’s job. Trying to use a single general-purpose chat model for both would either be inefficient (chat models are far larger and more expensive per call than embedding models) or simply wouldn’t work, since chat models aren’t built to output fixed-length numeric vectors suited for similarity search in the first place.

Why text-embedding-3-small specifically: embedding model choice is mostly a tradeoff between vector dimensionality, retrieval quality, and cost. The "small" variant produces a lower-dimensional vector than its larger sibling, which means less storage per chunk in the index and cheaper, faster similarity search, at a modest cost to retrieval precision on very nuanced or ambiguous queries. For a document set of 18 PDFs with fairly distinct topics per chunk, that precision tradeoff is negligible, and the cost and speed savings make it the sensible default rather than reaching for a larger embedding model that this dataset doesn't need.

Why the chat model choice mattered less here: the chat model’s job in a RAG pipeline is comparatively narrow. It isn’t being asked to reason over a huge amount of unstructured context or perform complex multi-step logic; it’s given a small set of retrieved chunks and asked to answer from them. That’s a task a smaller, faster, cheaper chat deployment handles well, which is why a lighter-weight model was chosen over a larger flagship one. The heavier lifting in this architecture is the retrieval step, not the generation step, and the model selection reflects that division of labor.

One naming note worth flagging: Azure OpenAI Service and Azure AI Foundry are not two separate products sitting side by side. Foundry is the current unified portal that Azure OpenAI now lives inside; a model “deployed in Foundry” is the same thing as a classic Azure OpenAI deployment, just managed through the newer interface. That distinction isn’t obvious the first time you go looking for “the Azure OpenAI resource” and don’t find a separate one.

Stage Three: Azure AI Search and the Vectorized Index

A standalone Azure AI Search resource was created on the Basic tier, the minimum tier that supports vector search. Using the built-in Import and vectorize data wizard, a data source (pointing at the blob container), a skillset (chunking plus embedding), an index, and an indexer were created in one flow, selecting the RAG scenario option specifically, since the goal was AI-generated answers grounded in text content rather than plain keyword search or complex visual interpretation.

Stage Four: The Indexing Failures, and What They Actually Meant

The first indexer run failed immediately with:

Web Api response status: 'NotFound'
DeploymentNotFound: The API deployment for this resource does not exist.

The skillset’s embedding step looked correct on paper: right deployment name, right model. The actual problem was the endpoint URL. The skillset had been auto-populated with the classic Azure OpenAI hostname format (<resource>.openai.azure.com), but the Foundry project's real endpoint used the newer unified format (<resource>.services.ai.azure.com). Two visually similar URLs, pointing at different places, and the deployment genuinely didn't exist at the one the skillset was calling.

Correcting the endpoint and switching the skill to rely on the Search service’s own managed identity (rather than an embedded API key) produced a second, different failure:

Web Api response status: 'Unauthorized' (401)
Access denied due to invalid subscription key or wrong API endpoint.

This one took longer to resolve. System-assigned identity was confirmed enabled on the Search service, and a Cognitive Services OpenAI User role was assigned to that identity on the Foundry resource, matching the pattern used for Cosmos DB access in an earlier build in this series. RBAC propagation delay was ruled out by waiting several minutes and retrying. The 401 persisted.

Rather than continuing to debug a single link in the chain against a fixed one-day timeline, the pragmatic call was made to fall back to API key authentication on the embedding skill specifically, keep the build moving, and document the Managed Identity attempt as an open gap rather than a completed pattern. With a valid key in place, the indexer run succeeded, and the index populated with the expected volume of chunked, vectorized content from the source PDFs.

This is worth being direct about for anyone building the same pattern: Managed Identity between Azure AI Search and Azure AI Foundry is documented as supported, but getting it working cleanly took longer than the time budget for this build allowed, and the exact cause (wrong role, a missing explicit auth identity block in the skill definition, or something else) wasn’t conclusively identified. The API key fallback works and is a legitimate interim choice, but it is a step back from the credential-less pattern this series has otherwise held to.

Stage Five: Wiring the Index to an Agent

Rather than attaching the knowledge base to a bare model in the playground, Foundry required creating an Agent: a saved construct combining the chat model, instructions, and a knowledge source into one callable entity. The knowledge base itself (built from the AI Search connection and index) was created as its own object first, then attached to the agent afterward.

The first attempt at chatting with this setup returned a hard error:

Access denied when connecting to the MCP server ... HTTP 403 Forbidden

The knowledge base was wired to a Managed Identity, but that identity had no RBAC role on the Search service itself yet, connecting an identity in the UI doesn’t retroactively grant it permissions. Assigning Search Index Data Reader to the correct managed identity on the Search resource’s Access Control (IAM) blade resolved the 403.

Stage Six: Two More Gaps, Both Behavioral Rather Than Configuration Errors

With the 403 resolved, the agent chat returned “I don’t have any answer for you” for every query. This looked like another broken connection, but the actual cause was upstream: the index itself was still empty at that point in the build (the indexing failures from Stage Four hadn’t been resolved yet in the sequence this was actually built). Once the index was genuinely populated, retesting the same queries returned real, correctly grounded content pulled verbatim from the source slides.

The second gap surfaced after that: asking the agent a direct question sometimes returned a fluent, plausible-sounding answer that did not match the actual course material at all; specifically, a generic six-category steganography taxonomy that doesn’t appear anywhere in the indexed slides. The trace log showed the agent had access to the knowledge base as a callable tool (mcp_list_tools appeared in the trace) but had simply chosen not to invoke it for that particular question, and fell back to general model knowledge instead.

This is the most important lesson from the whole build: connecting a knowledge source to an agent makes retrieval available, it does not make retrieval mandatory. The model decides, turn by turn, whether calling the retrieval tool is worth it, unless explicitly told otherwise.

The fix was adding explicit instructions to the agent’s configuration:

You are an assistant that answers questions about digital forensics course material.
For every user question, you MUST first search the connected knowledge base before answering.
Only answer using information retrieved from the knowledge base.
If the knowledge base returns no relevant results, say so explicitly rather than answering from general knowledge.

A fresh chat session after saving this returned an answer matching the actual indexed slide content precisely, including the exact category list and structure from the source deck, rather than a plausible-sounding substitute.

What’s Actually Happening Behind the Scenes

Stripped of the portal clicks, this build surfaced a distinction that’s easy to miss going in: a RAG pipeline has a retrieval half and a generation half, and each one can fail silently in a way that looks like the other. An empty index and an ungrounded agent produce different symptoms (a flat “no answer” versus a confident wrong answer) but both come from the same underlying issue, the retrieval step not actually running, for different reasons at different stages.

It’s also worth naming plainly that this build did not end in a fully credential-less state. The embedding skill runs on an API key, not Managed Identity, after Managed Identity produced a persistent, unresolved 401 within the available time. That’s a real gap against the zero-trust pattern this series has otherwise maintained, not a detail to bury in a footnote.

The output mode chosen, Extractive, was a deliberate one for proving grounding clearly (the responses come back close to verbatim from the source text, which makes it obvious the model is pulling from the actual documents rather than paraphrasing from memory), but it is not what a polished end-user chatbot would ship with. Slide numbers, copyright boilerplate, and raw formatting come through in the answers. A production version of this would likely use a more generative output mode paired with tighter retrieval instructions to produce cleaner, still-grounded responses.

Challenges and Lessons Learned

The Azure OpenAI endpoint format mismatch (openai.azure.com versus services.ai.azure.com) was the first real trap, two URLs that look like reasonable guesses for the same resource, only one of which is correct for a Foundry-provisioned deployment.

The Managed Identity 401 between AI Search and Foundry was the single largest time cost in this build and was not fully resolved. Documenting an unresolved gap honestly, rather than quietly switching to a key and not mentioning it, is more useful to anyone following this pattern than a report implying full success.

The 403 on the agent’s knowledge base connection was a cleaner, faster fix: connecting an identity in a UI panel is not the same as granting that identity a role, the same lesson as the Cosmos DB data-plane RBAC gap from an earlier build in this series, just in a different service.

The most conceptually important lesson was the last one: an agent with a connected knowledge source will not necessarily use it. Retrieval has to be made mandatory through explicit instructions, or the agent will sometimes answer fluently and incorrectly from general model knowledge instead, and nothing in the UI warns you this is happening until you check the response against the actual source material.

Key Takeaways

• RAG has two halves, retrieval and generation, and each can fail silently in a way that mimics the other; an empty index and an ungrounded agent both look like “it’s not working” for different underlying reasons.
• Azure AI Foundry and Azure OpenAI Service are the same underlying resource; Foundry is the current management layer, not a separate parallel product.
• Foundry-provisioned endpoints use a different hostname format than classic Azure OpenAI resources, and skillset wizards can auto-populate the wrong one.
• Connecting a Managed Identity to a resource in a configuration panel does not grant that identity any permissions; the RBAC role assignment is a separate, required step.
• An agent with a connected knowledge base treats retrieval as an optional tool call by default, not a guarantee; forcing mandatory retrieval requires explicit instructions.
• Not every credential-less goal gets fully achieved in a single build, and documenting the fallback (API key over Managed Identity, in this case) honestly is more valuable than hiding it.