The BA Blind Spot in Every AI Project

How understanding AI Architecture can help Business Analysts drive AI projects

I am a Business Analyst. But I am also an AI enthusiast.

Over the past year, I have been working on several AI initiatives at Spectrum. The more I worked on them, the more I wanted to go deeper, not just understand the outputs, not just write the user stories, I wanted to understand the architecture and what's happening inside these systems, so I can ask better questions as a Business Analyst.

So this past weekend, I sat down and mapped it out. From scratch. For myself.

I built a reference architecture for an AI assistant or chatbot because that is one of the most common AI initiatives you will see in any enterprise right now. Almost every company is either building one or planning one, and as I was simplifying things for my own understanding, I realized something. I had created a framework that was too useful to keep to myself.

So here it is.

Why does this matter to you as a BA?

Because architecture shapes everything downstream. The requirements you gather, the edge cases you identify, the risks you flag, and the questions you ask in design reviews. If you do not have a mental model of how these systems work, you will always be one step behind in the conversation. This article gives you that mental model.

Before breaking down each component, here is the end-to-end flow the diagram below represents. Seven steps. Everything else is detail.

The Full Flow

1. The user sends a message through the chatbot app
2. The AI agent receives it and decides what to do
3. If it needs knowledge, it searches the vector database and feeds results plus the question to the LLM (this is RAG)
4. If it needs to take action, it calls external tools via MCP
5. If it needs both, it does both and combines everything
6. The LLM generates the final response
7. The agent sends it back through the chatbot app to the user

Two rules hold across the entire flow:

1. The agent is always the central hub. Nothing bypasses it.
2. Every connection is bidirectional. Requests go down, results come back up.

Understanding every component allows you not just to be a part of the conversation, but also to contribute to it.

Now let’s go deeper into each piece.

1) Chatbot App

This is the only part of the architecture the user ever sees.

It handles the UI, authentication, session management, message routing, and integration with the backend.

It is worth saying clearly: the chatbot app is not where the intelligence lives; it is a delivery mechanism.

Think of it as the front door. What matters is what happens once you walk through it.

2) AI Agent

This is the orchestration layer that sits on top of the LLM. It is the brain that decides which path to take for each part of a user’s request. It receives the user’s question, determines what is needed, fetches context from the vector database if relevant, packages that context plus the question into a prompt, sends it to the LLM, receives the response, and delivers it back.

The agent is why nothing in this architecture talks directly to anything else. The LLM and the vector database never communicate with each other. The LLM and external tools never communicate with each other. The agent always sits in the middle.

A system can have one agent or multiple agents.

You need multiple agents when:

1. Different domains require different knowledge. A returns agent needs product policy docs. A billing agent needs payment system access.
2. Different tools are required. One agent searches your CRM, another books meetings. Giving one agent access to everything makes it unpredictable.
3. Different guardrails apply. Your customer-facing agent has strict tone and content rules. Your internal ops agent has broader permissions. One set of rules cannot govern both.
4. Different levels of autonomy are needed. Processing a refund may require a human in the loop. Answering an FAQ does not.

A single agent works fine when:

1. The scope is narrow
2. The knowledge lives in one domain
3. The tools are limited

Most systems start with one agent and split later when complexity grows.

3) LLM (Large Language Model)

Think GPT-4, Claude, Llama, Gemini. This is the reasoning engine that generates responses by predicting what text should come next, based on patterns learned during training across billions of documents.

Here is what most people get wrong about LLMs:

1. They are not databases
2. They do not store and retrieve information
3. They do not look things up
4. They generate responses by predicting what text should come next, based on patterns learned during training

Imagine a student who spent years reading thousands of textbooks, then sat down to take an exam without any of those books in front of them. They’re recalling and reasoning from what they absorbed. Sometimes they get things right, sometimes they confidently get things wrong. That confident wrongness is what people call hallucination. And it is a structural feature of how LLMs work, not a bug that will be patched away.

One more thing worth knowing: the LLM has no idea where its input came from; it receives text. Whether that text was retrieved from a vector database, pulled from a PDF, or typed by hand, it does not matter to the LLM; it just generates a response. Everything else is handled by the agent and MCP.

4) RAG (Retrieval Augmented Generation)

Unlike everything else in this list, RAG is not a component; it is a technique.

It describes the pattern of retrieving relevant documents from a knowledge source and augmenting the LLM’s prompt with those documents so it generates more accurate, grounded answers.

There are two steps to RAG:

1. Retrieve — First, relevant documents are pulled from a knowledge source
2. Augment — Second, the documents pulled are added to the LLM’s prompt so it generates more accurate, grounded answers

RAG exists because LLMs, on their own, only know what they learned during training. They have no access to your company’s internal documentation, your product catalog, your policy library, or anything that wasn’t in their training data. RAG solves this by pairing the LLM with an actual knowledge base, and the agent is what makes the retrieval happen.

5) Vector Database

This is where your organization’s knowledge lives, but it does not store documents the way a file system does.

Before any user ever sends a message, every document in your knowledge base is converted into a mathematical representation called an embedding. This is a numerical vector that captures the meaning of the text, not just the words. The conversion happens during an initial ingestion process, then on a schedule whenever new content is added. When a user asks a question, the agent converts that question into a vector too and searches for semantically similar content.

This is how the system can return a document about “refund timelines” in response to a question about “how long does it take to get my money back,” even though none of those exact words appear in the document.

Common vector databases: Pinecone, pgvector.

Important: There are actually two kinds of data sources in most AI architectures.

The agent decides which one to query based on the user's needs and can query both in a single interaction.

For example, when a customer asks: “What is your return policy and where is my order?” The agent uses RAG to retrieve the policy documentation. It uses a tool call to look up that specific customer’s order. It combines both into one seamless response. The user has no idea that two completely different systems were just queried.

6) MCP (Model Context Protocol)

This is the protocol that connects the agent to external tools and live systems like Slack, Jira, or your internal databases. It is an open standard created by Anthropic that defines how AI agents connect to external tools and data sources.

Think of MCP as USB for AI. Before USB, every device had its own proprietary connector — a different cable for every peripheral. USB standardized the interface so any device could connect to any port. MCP does the same thing for AI tool integrations.

Without MCP, you would write custom integration code for every tool your agent needs. One connector for your database, one for your CRM, one for Slack, each with its own format and conventions. With MCP, any MCP-compatible tool can be discovered and used the same way, regardless of what it does underneath.

There are three distinct pieces:

1) MCP Client

• Lives inside the AI agent
• Makes requests to tools
• You never build this. It is built into modern agent frameworks like LangChain, Amazon Bedrock, and Kiro
• If building a fully custom agent, you add it with a few lines of code using an MCP SDK

2) MCP Protocol

• The communication standard, rules, and format for requests and responses
• You never build this. It is an open standard, like HTTP
• Everyone uses the same one. That is the entire point.

3) MCP Server

• A lightweight wrapper that makes a specific tool accessible through the standard protocol
• This is the only piece you might actually build
• For common tools like Slack, GitHub, Jira, and Postgres, hundreds of prebuilt servers already exist. Configure, don’t code.
• For proprietary internal tools, you wrap your internal API with the MCP SDK. A relatively small amount of code for a significant unlock.

Think of it this way: the MCP client is your phone. The MCP protocol is the phone network, the rules for how calls work. The MCP server is the person you are calling.

The practical summary:

What Are We Actually Building Here?

One of the most useful questions a BA can ask on an AI initiative: what are we building, and what are we just configuring?

Understanding this split changes the questions you ask. Instead of “how long will it take to build the AI,” you start asking:

• Which parts are we buying?
• Which parts are we configuring?
• Which parts are we actually building?
• What does that mean for timeline, vendor risk, and maintenance?

And this is exactly why the BA’s role on an AI project is so critical. When there is less to build, there is more to decide. The questions around governance, data ownership, compliance, business rules, and failure handling do not disappear because the technical effort is lower. They become the entire job.

The Scenario: An AI Assistant for Insurance Customer Service

Imagine your company is building an AI assistant for insurance customers. A customer opens the app and asks:

“Am I covered for physiotherapy after my knee surgery, and what is the status of my last claim?”

Simple question from the customer’s perspective. But look at what needs to happen underneath:

1. The agent needs to retrieve the customer’s policy documentation to answer the coverage question (RAG, vector database)
2. The agent needs to look up that customer’s specific claim record in a live operational system (tool call via MCP)
3. The LLM needs to combine both answers into one clear, accurate response
4. The whole thing needs to happen without the customer knowing two completely different systems were just queried

This is not a simple FAQ bot. This is a multi-source, multi-system interaction. And every decision in this architecture has BA implications.

The Questions a BA Should Be Asking

Most BA analysis on AI projects stays at the surface level.

What should the chatbot answer? What tone should it use?

Elicitation is where you go and confirm it with the people who actually know. These are the conversations to have with your business stakeholders, subject matter experts, and engineering team. Understanding an AI architecture and how all the components in it flow, help Business Analysts ask more effective questions during elicitation. Below are the list of AI elicitation questions for each stakeholder group:

With Business Stakeholders

These are the people who own the outcomes. They need to understand what they are committing to.

1. What does success look like for this assistant in the first 90 days? Are we measuring containment rate, customer satisfaction, call deflection?
2. What is the risk appetite for incorrect answers? In insurance, a wrong coverage answer can have legal consequences. Who owns that risk?
3. Which customer segments will use this assistant? Are there accessibility or language requirements we need to account for?
4. What is the escalation path when something goes wrong? Who do we call when the assistant gives a customer bad information about their claim?
5. Is there a regulatory or legal review process for the responses the LLM generates?

With Subject Matter Experts

SMEs are your knowledge owners. They hold the information the vector database needs.

1. Walk me through how a customer would typically ask about their coverage. What are the most common phrasings and edge cases?
2. Which documents contain the authoritative answer to a coverage question? Are there conflicts between documents?
3. What questions do your human agents struggle with today? Those are likely the same ones the AI assistant will struggle with.
4. What information should never be shared with a customer through an automated channel?
5. How frequently do policies, products, or guidelines change? Who sends the update, and to whom?

With the Engineering Team

These conversations are about translating business needs into system constraints.

1. How many agents are we building, and what is each one responsible for?
2. How is the agent deciding whether to retrieve from the vector database, call a live system, or both? Can you walk me through the decision logic?
3. What happens when the vector database returns a low confidence match? Does the agent still pass it to the LLM, or does it flag it?
4. How are MCP connections authenticated, and what audit logging exists for every tool call the agent makes?
5. What is the latency expectation for a response? If the agent needs to query two systems and combine the results, what is the acceptable wait time?
6. How will we test the LLM’s responses before going live? What does the UAT process look like for an AI system?

With Compliance and Legal

On any AI initiative in a regulated industry, this conversation is not optional.

1. Are there specific disclosures required when a customer is interacting with an AI rather than a human?
2. What data retention policies apply to conversation history?
3. If a customer makes a decision based on an incorrect AI response, what is our liability position?
4. Does this assistant need to be explainable? Can we show a regulator why it gave a specific answer?

The User Stories That Come Out of This

Once you have asked these questions, your user stories stop being vague and start being precise. Compare these two versions:

Before understanding the architecture: As a customer, I want to ask questions about my policy so that I get helpful answers.

After understanding the architecture: As a customer, I want to ask whether a specific treatment is covered under my policy so that I can make an informed decision before my appointment.

Acceptance criteria:

1. The assistant retrieves the relevant section of the customer’s active policy document from the knowledge base
2. If the policy document does not contain a clear answer, the assistant tells the customer it cannot confirm coverage and provides the number for the claims team
3. The response cites the policy section it is drawing from
4. If the customer’s policy has been updated in the last 30 days, the assistant uses the most recent version
5. Response is delivered within 4 seconds

That is the difference architecture awareness makes. You are not just describing what the user wants. You are describing what the system needs to do, where the data comes from, what the failure conditions are, and how you will know it worked.

Save This. Use It.

I put this together for myself and then realized it was too useful to keep private.

If you are a BA working on any AI initiative, this is your framework. Not for building the system. For asking the questions that make the system worth building.

The engineers will handle the architecture. Your job is to make sure the right questions get asked before a single line of code gets written.

That is what this weekend was about for me.

A Saturday well spent.

About The Good BA This article is part of The Good BA newsletter, where I share insights, frameworks, and real-world experiences from the world of Business Analysis.

👆Subscribe to my LinkedIn Bi-weekly newsletter: **The Good BA!**

TheGoodBA #ProductOwner #BusinessAnalyst #ITBusinessAnalyst #BusinessSystemsAnalyst #BusinessAnalysis #AI #AIAgent #RAG #AIArchitecture #AIAssistant #LLM #MCP #AIChatbot