Photo by Chor Tsang on Unsplash

The Real Problem in Enterprise AI Is Context Selection

An enterprise team connects a language model to every internal source it owns. Documents, tickets, wikis, CRM records, and dashboards all feed the same pipeline. The demo looks good and promising. Three weeks into production, the answers are inconsistent, occasionally wrong, and sometimes leak information a user should never see.

The retriever works. The model works. The system still fails.

The failure sits in a layer most teams never designed on purpose. The real problem in enterprise AI is context selection.

Context selection is the discipline of deciding what information reaches the model, from which source, by which method, and under what constraints. RAG is where this conversation starts. It is not where it ends.

Why Context Selection Is the Unsolved Problem

Most enterprise AI effort goes into two places. Teams tune the model, then tune retrieval. Both matter. Neither addresses the harder question of what should reach the model in the first place.

A model can only reason over what it receives. Retrieval decides what it receives. But retrieval alone assumes every request is a similarity problem, and enterprise requests rarely are.

Context selection sits above retrieval as a routing and control layer.

It answers a different set of questions. Which source holds the truth for this query? Which method fits the shape of the request? Who is allowed to see the result? Whether the answer can be trusted enough to return at all.

These are design decisions, not tuning knobs. A team that skips them ships a pipeline, not a system.

In short: the model reasons, retrieval fetches, but context selection governs the whole exchange.

Context Routing: The Decision Before Retrieval

Every enterprise query has a natural home. A question about a signed policy lives in documents. A question about an unpaid invoice lives in a transactional system. A question about quarterly revenue lives in a data warehouse.

Context routing is the layer that reads the request and picks the right path before any retrieval runs.

Without routing, teams default to one mechanism for everything. Usually that mechanism is a vector search. It returns a plausible match for a numerical question that needed SQL, or a stale document for a question that needed a live API call.

The result is a system that answers everything and gets much of it subtly wrong.

How routing works in practice

A routing layer classifies the query, then assigns it a source and a method. The steps are straightforward:

• Classify intent: lookup, calculation, action, or synthesis.
• Identify the authoritative source for that intent.
• Select the retrieval or query method that fits.
• Pass control to the chosen path, not the default one.

The guardrail here is honesty about classification confidence. When the router cannot confidently place a query, the safe move is a clarification prompt or a fallback, not a guess. A confident misroute produces a confident wrong answer, which is the most expensive failure of all.

Permissions and Access Control: Context With Boundaries

In a demo, every user sees everything the index contains. In an enterprise, that same design is a data breach waiting for an audit.

RAG Inside the Full System

genai-system/
├── user-intent-understanding/
├── context-engineering/
│   ├── rag/
│   │   ├── vector-search/
│   │   ├── hybrid-search/
│   │   ├── reranking/
│   │   ├── graph-rag/
│   │   ├── hierarchical-rag/
│   │   └── contextual-retrieval/
│   │
│   ├── tool-based-context/
│   │   ├── SQL/
│   │   ├── APIs/
│   │   ├── search/
│   │   └── file-readers/
│   │
│   ├── memory/
│   ├── access-control/
│   └── context-compression/
│
├── model-orchestration/
├── evaluation/
├── observability/
└── governance/

RAG is positioned alongside SQL, APIs, search, and code analysis tools within context engineering. This layer of context engineering is itself situated next to evaluation, governance, and observability, forming layers within the entire system.

Access control is not a feature added after retrieval. It is part of context selection itself.

The model should never receive a passage the requesting user is not entitled to see. That means filtering happens before ranking, not after. A document the user cannot access should never enter the candidate set, because anything in the context window can surface in the answer.

Enterprise context selection enforces permissions at several levels:

• Document level: who can access a given file or record.
• Chunk level: which sections within a document are visible to a role.
• Source level: which systems a user or agent may query at all.

Each level closes a gap the naive pipeline leaves open.

The guardrail is traceability. Every answer should link back to the sources that produced it, so a reviewer can confirm both the accuracy and the authorization. An audit log that records what was retrieved, for whom, and why turns a black box into a governable system.

In short: context selection without access control is not enterprise-ready, no matter how good the retrieval is.

Retrieval Fit: Matching Method to Query Type

Retrieval is not one technique. Treating it as one is the quiet cause of many stalled projects.

Different questions demand different retrieval methods.

A question with a single factual answer needs precision. A broad research question needs recall across many sources. A question that spans several documents needs synthesis, which flat vector search handles poorly. Graph or hierarchical retrieval fits that case far better.

Retrieval fit is the practice of matching the method to the shape of the question.

Consider the failure it prevents. Ask about a notice period in a B2B contract, and a naive retriever returns the clause “the notice period is six days.” It drops the heading above it stating the clause applies only to permanent employees. The passage is similar. The answer is wrong.

Contextual retrieval, structured chunking, and reranking exist to close this exact gap. Anthropic published research showing that contextual retrieval, which enriches each chunk with surrounding document context before indexing, reduces failed retrievals by a measurable margin.

A short checklist for retrieval fit

• Single fact required: favor precision and reranking.
• Broad coverage required: favor higher recall with filtering.
• Cross-document synthesis required: favor graph or hierarchical retrieval.
• Small corpus, one-time question: favor direct long-context analysis over a full pipeline.

The guardrail is restraint. More retrieved context is not better context, because models weight information unevenly and lose detail buried in the middle of a long window. A precise, well-ordered set of passages outperforms a large, noisy one.

Evaluation: The Mechanism That Catches Failure

A system without evaluation fails silently. It produces confident wrong answers, and no one notices until a person catches the mistake by hand.

Evaluation is the part of context selection that measures whether the earlier decisions worked.

It answers two separate questions. Did the system faithfully retrieve the right context, and did the model use it? A good answer built on the wrong context is luck. A wrong answer built on the right context is a generation problem. Evaluation tells the two apart.

Frameworks such as RAGAS define metrics for exactly this: faithfulness, answer relevancy, context precision, and context recall. The first two measure answer quality. The last two measure retrieval quality, which is where most failures actually originate.

Without these metrics, a team optimizes blind.

The guardrail is continuous measurement, not a one-time test. Documents change, user behavior shifts, and a pipeline that passed last quarter can degrade quietly this quarter. Evaluation running in production is the difference between a system you trust and a system you hope about.

In short: evaluation is not the final step, it is the feedback loop that keeps context selection honest.

Common Mistakes in Enterprise Context Selection

Teams tend to repeat the same errors. Each one traces back to treating context selection as an afterthought rather than the core design.

• One mechanism for every query. Routing everything through vector search ignores the shape of the question and the location of the truth.
• Permissions bolted on late. Access control added after retrieval leaves a window where restricted content can reach the model.
• Retrieval treated as a single setting. A fixed chunk size and a fixed top-k value cannot serve precision, recall, and synthesis at once.
• No evaluation in production. Without measurement, failures stay invisible until a user or an auditor finds them.
• More context assumed to be better. Oversized context windows dilute attention and lower precision instead of raising it.

The fix for all five is the same. Design context selection as a deliberate layer, with routing, access control, retrieval fit, and evaluation as named responsibilities.

Build the Context Layer First

Enterprise AI reliability depends not on a more advanced model or quicker vector storage, but on deliberate decisions about what data is fed to the model and the rules governing that process. Context selection is that decision layer.

Routing sends each query to the right source and method. Access control keeps every answer inside its permission boundary. Retrieval fit matches technique to question type. Evaluation confirms the whole chain held.

Start with one process that already matters to the business. Map its queries to sources, define the permissions, choose retrieval methods that fit, and wire in evaluation before you scale. Then expand that pattern outward.

The teams that treat context selection as core engineering build enterprise AI that holds up under real use. The teams that treat it as plumbing keep rebuilding the same unreliable demo.