Hindsight: A Knowledge Graph Layer for Navigating Large Codebases

How graph-augmented retrieval outperforms keyword search for multi-file code understanding, and where it still falls short

Article by Priyamvadha Balakrishnan and Vanshika Agrawal.

The problem: code lives between the files

Ask a useful question about a large repository, such as why a test fails, what a change will break, or how a feature is actually implemented, and the answer is almost never in one place. Functionality is spread across files, modules, and dependency chains. Understanding it requires reasoning over that structure, not retrieving a single relevant snippet.

Most retrieval systems treat code as flat text, rank chunks by surface similarity, and miss the relationships (function calls, imports, dependency paths) that connect the real answer across files. A keyword search can find the file where a question starts, but the evidence that resolves it usually sits one or two hops away, in a utility module or a caller the query never mentions by name.

HindSight is designed to close that gap. It pairs BM25 lexical retrieval with a repository knowledge graph: initial candidates are expanded through graph traversal over calls, imports, and dependencies to assemble richer, multi-file context before the model generates an answer.

The central finding, stated directly: graph-guided retrieval outperforms BM25 on broad, cross-file queries and on top-1 file localization, precisely the cases where keyword search provides insufficient signal. On deep, lexically-anchored questions, BM25 remains competitive. Across all conditions, the performance ceiling is determined not by how well the model generates but by how completely the retrieval stage recovers the relevant context. To situate this finding, it is useful to survey what prior work has established.

What others have established

Code question answering has steadily climbed the ladder of difficulty. Early benchmarks like CodeQA scored models on individual functions and snippets. Newer ones, including DeepCodeBench, SpiderCodeQA, and CoReQA, demand realistic multi-file reasoning grounded in pull requests, repository semantics, and GitHub issues. The trajectory is clear: the field is moving toward structurally-grounded, repository-level evaluation.

The retrieval literature explains why this is hard. CoIR and DependEval both show that purely lexical or embedding-based retrieval struggles when the answer depends on repository structure, such as dependency paths and import chains, rather than text overlap. Most pipelines still operate over flat text chunks and rank by similarity, which is precisely the wrong primitive for structural questions.

A handful of systems bring structure into the loop. LocAgent, the closest prior work, uses a heterogeneous code graph for file localization. RepoGraph and Code Graph Model (CGM) represent repositories through dependency and AST structures. HindSight’s distinction is where it places the structure: rather than treating localization as a separate stage, it folds structural signals directly into retrieval through intent-guided traversal. This work focuses on the upstream bottleneck of retrieval and localization quality, rather than end-to-end issue resolution.

The method: decompose, classify, then traverse

Before building anything, we benchmarked the alternatives under one unified setup: the same repository snapshots, chunking, prompt templates, and context budget for BM25, CodeRankEmbed (a dense retriever), and LocAgent. Holding everything else constant isolates retrieval quality as the variable that moves results.

We organize developer questions into three types, because each places different demands on retrieval:

• Leading: asked before a change, to anticipate impact (“what depends on this API?”)
• Lagging: asked after a failure, to trace a symptom to its cause (“what code path produced this bug?”)
• Exploratory: asked to understand unfamiliar code with no change in flight (“how does this mechanism work?”)

HindSight builds on LocAgent’s code graph, a directed, heterogeneous graph whose nodes are directories, files, classes, and functions, connected by contain, import, invoke, and inherit edges. On top of LocAgent's retrieval loop, HindSight adds two steps that change how the agent allocates its search budget.

Query decomposition. Real questions bundle several retrieval intents into one sentence. “Which helper function converts object-detection outputs into filtered boxes, scores, and labels in the integration tests?” implicitly asks the system to locate the test files, find the output-processing logic, and link them. HindSight breaks the original query into focused sub-questions before retrieval begins. Those sub-questions give BM25 sharper lexical anchors and give graph traversal more targeted entry points into the AST indexes.

Intent classification. Each query is then labeled by an LLM classifier as leading, lagging, or exploratory, and traversal adapts to the label. Leading queries follow outgoing invoke and import edges to surface the downstream dependencies of a planned change. Lagging queries follow incoming edges back toward a root cause. Exploratory queries either expand broadly across files or descend into a single file's class-and-function hierarchy via the AST index, depending on whether the question is broad or deep. This intent-conditioned traversal is the central design choice that separates HindSight from lexical-first baselines.

The HindSight pipeline. A repository instance is indexed via LocAgent AST and BM25; queries are decomposed and classified by intent; graph traversal produces localized outputs evaluated on Fact Recall and Answer Quality across DeepCodeBench and LocBench.

Step-by-step walkthrough of HindSight’s retrieval on the query “Why is the CUDA batching equivalence test skipped?” BM25 retrieves the test file as the initial candidate; graph traversal then surfaces gpu_utils.py (CUDA device conditions) and lightglue_model.py (model implementation) via import and invoke edges, assembling the multi-file context needed to answer the question. The answer was never in a single file.

How we measured it

We evaluate on two complementary public datasets so that both stages of code intelligence (finding the right context and producing the right answer) are tested.

DeepCodeBench contributes 1,144 question-and-answer pairs from real open-source repositories and is overwhelmingly exploratory (96%); it measures answer quality. LocBench contributes 560 issue-to-code localization tasks (242 bug reports, 150 feature requests, 139 performance issues, and 29 security vulnerabilities), split roughly evenly between leading (48%) and lagging (51%) intents; it measures whether the system finds the right file or function. Together they cover the full query space.

Intent distribution across DeepCodeBench (overwhelmingly exploratory, 96%) and LocBench (evenly split between leading, 48%, and lagging, 51%). Together the two datasets cover the full range of developer query types.

Evaluation baselines ordered by increasing access to repository evidence, from Limited Context (lower bound) to Full PR Context (upper bound). All conditions use identical repository snapshots, chunking, and context budgets.

Baselines run along a ladder of increasing access to evidence, from Limited Context (the repository name only, a lower bound), to BM25, to CodeRankEmbed, to LocAgent, and finally to Full PR context (the issue or PR text and anchored code, an upper bound). We report three metrics. Fact Recall has an LLM judge check whether each gold fact appears in the generated answer. Answer Quality is scored from 1 to 10 by an LLM judge on accuracy, completeness, relevance, and clarity. Acc@k measures whether the correct file, module, or function lands in the top-k predictions.

We ran four models (Llama-3.1–8B, Qwen3–8B, Qwen3–32B, and GPT-5.3-codex) and concentrate the analysis on Qwen3–32B as a competitive open-source backbone, with GPT-5.3-codex as the performance ceiling. This ceiling is significant for two reasons: it bounds what the pipeline can achieve under any retrieval strategy, and its high-quality outputs serve as labeled data for downstream fine-tuning. Critically, the gap between GPT-5.3-codex and the open-source models holds steady across every retrieval strategy, indicating a model-capacity gap rather than a retrieval gap.

What we found

Retrieval matters enormously; structure matters selectively

Any form of retrieval is transformative. For Qwen3–32B, Fact Recall climbs from 0.097 with no retrieval to 0.520 with BM25. Multi-file repositories cannot be understood from the repository name alone. On overall Fact Recall, BM25 leads (0.520), with CodeRankEmbed (0.509) and HindSight v1 (0.482) close behind. The first two numbers carry their own lesson: dense embeddings add almost nothing over a well-tuned sparse retriever for repository QA. GPT-5.3-codex sets the strongest ceiling under every retrieval condition (0.595 Fact Recall with BM25, versus 0.505 for Qwen3–32B), and that margin is consistent regardless of retriever, confirming a capacity gap rather than a retrieval gap.

Fact Recall for Limited Context, BM25, CodeRankEmbed across 4 models (Qwen3–8b, Llama-3.1–8B, Qwen3–32B, GPT-5.3-codex) and Fact Recall for LocAgent for Qwen3–32b. Any form of retrieval is transformative; the gap between GPT-5.3-codex and open-source models is consistent across every retrieval strategy, indicating a model-capacity gap rather than a retrieval gap.

Graph retrieval wins where lexical anchors are weakest

HindSight’s advantages are not distributed evenly; they concentrate precisely where the system design predicts. It leads on broad-scope questions (0.493 Fact Recall vs. BM25’s 0.467), on non-searchable questions, and on top-1 file localization (Acc@1 of 0.535 vs. BM25’s 0.454, a gain of 8.1 points, with adapted LocAgent trailing at 0.423). The single largest gain comes on queries with no location hints, where file-level Acc@1 reaches 0.517 against BM25’s 0.388, a 12.9-point improvement. When keyword search lacks sufficient lexical signal, query decomposition reconstructs the retrieval anchors that BM25 alone cannot establish. The same pattern holds on non-core code paths such as utilities, tests, and peripheral modules, where HindSight reaches file Acc@1 of 0.550 against BM25’s 0.439.

Localization Acc@k at file, module, and function levels across query categories (core, non-core, broad, deep, with/without location hints) for BM25, HindSight v1, and adapted LocAgent. HindSight leads at file-level Acc@1; BM25 leads at finer granularity.

Localization does not automatically become fact recall

Finding the right file is not the same as ranking the right entity within it. HindSight surfaces the correct file but does not yet order entities within it as precisely as BM25: at module level, BM25 leads on Acc@5 (0.436 vs. 0.363), and the same holds at function level (0.361 vs. 0.283). Query intent shapes the error pattern. Leading queries are easier to localize at the file level (Acc@1 0.535 vs. 0.467 for lagging), because planning questions name features directly. Lagging queries perform better at finer granularity (function Acc@5 0.340 vs. 0.260), because bug reports describe symptoms that map onto specific methods.

Localization Acc@k on LocBench and DeepCodeBench. HindSight v1 with GPT-5.3-codex achieves the strongest file-level results; BM25 leads at module and function granularity for open-source models.

Acc@k by query category bar chart: File-, module-, and function-level Acc@k for Qwen3–32B by intent (leading, lagging, exploratory). Leading queries localize more accurately at the file level; lagging queries perform better at finer granularity, as bug reports map more directly onto specific functions.

The gap to Full Context is a completeness problem, not a style problem

The LLM-judge dimensions localize the remaining gap precisely. Clarity and Relevance are near-saturated even without retrieval. For Qwen3–32B, Clarity rises only marginally, from 8.160 to 8.735 with BM25. Accuracy and Completeness are what retrieval actually shifts: Accuracy rises from 3.860 to 6.696, and Completeness from 3.931 to 6.623. Models produce fluent answers even when poorly grounded; the deficiency is factual coverage, not linguistic quality. A second constraint applies: open-source models at 8B to 32B trail GPT-5.3-codex on accuracy and completeness regardless of retrieval quality, with GPT-5.3-codex reaching 8.142 Accuracy under full context. Closing that gap without depending on a proprietary API is a practical requirement for deployment at scale.

Quality dimensions line chart: Answer quality dimensions (Accuracy, Completeness, Relevance, Clarity) by retrieval condition for all four models. Clarity and Relevance saturate early; Accuracy and Completeness are what retrieval moves, and what separates GPT-5.3-codex from open-source models.

What it adds up to

The structural hypothesis holds, but only conditionally. Graph signals provide the strongest benefit for broad, ambiguous, and non-searchable queries, precisely the cases where surface similarity is weakest. For queries with a strong lexical fingerprint, a well-tuned sparse retriever is difficult to surpass, and that finding is worth stating plainly.

Two bottlenecks remain, and identifying them precisely points toward the appropriate remedies. The first is granularity: HindSight reaches the correct file, but ordering entities within it is a reranking problem, not a graph-construction problem. The second is completeness: the gap to Full Context is real and large, driven by accuracy and completeness rather than by how the answer reads. Underlying both bottlenecks is model capacity, which the results show is independent of retrieval strategy and points directly to the value of distillation.

Where this goes next

Four concrete directions follow from these findings.

First, a test-linkage index that connects test functions to the code they exercise, improving coverage on non-core and lagging queries where failures trace through test infrastructure. Alongside this, broad exploratory queries will be served by a RepoGraph symbol index (a repository-wide symbol layer that enables file-spanning traversal for questions not anchored to a specific code location), once the core pipeline is validated on the current graph. Second, an intent-aware reranker adapted from Code Graph Model, ranking retrieved entities by query intent and decomposed sub-questions to close the module-level and function-level gap without giving up the file-level advantage. Third, a per-intent traversal policy that uses file-level expansion for leading queries and function-level descent for lagging queries, refined for the exploratory and non-searchable cases that remain furthest from the upper bound.

Fourth, and most consequential, distillation from GPT-5.3-codex. The complete pipeline (test-linkage index, RepoGraph symbol index, intent-aware reranker, and per-intent traversal) will run with GPT-5.3-codex as the localization model, responsible for decomposition, classification, and graph traversal. Its highest-scoring localization outputs, filtered by Acc@k thresholds at the file, module, and function level as well as by Fact Recall, become a curated training set for fine-tuning a smaller open-source model such as Qwen3–8B or Llama-3.1–8B. The goal is an open-source localization backbone that matches GPT-5.3-codex retrieval quality with no API dependence: no per-query cost, no data leaving the organization, and no model-availability risk. For large proprietary codebases, that is the path to deployment at industrial scale.

The contribution is straightforward to state. Hybrid retrieval with intent-guided graph traversal improves multi-file code understanding, with the clearest gains on the broad, structurally complex queries that defeat keyword search. The honest counterpoint is equally clear: BM25 remains strong wherever a query carries explicit identifiers, and ranking within files and closing the model-capacity gap are the two problems left to solve. Addressing both will make code intelligence reliable across the full range of developer work: proactive impact analysis before a change, root-cause tracing after a failure, and onboarding onto unfamiliar code at scale.

HindSight was developed at the University of Massachusetts Amherst in collaboration with 99P Labs and Honda Research Institute. The authors thank their course staff and project mentors, and acknowledge the the computational resources behind these experiments.

Research team: Vanshika Agrawal, Priyamvadha Balakrishnan, Mustafa Ali, and Sheng Kai Wen. PhD mentor: Anshita Gupta. Industry mentors: Ryan Lingo and Rajeev Chhajer.