Graphifyy — A Good Tool for a Problem We Didn’t Have

Three ways to wire a code knowledge graph into a coding agent, three measurements, and the one number that explained all of them

There is a particular kind of engineering mistake that never feels like a mistake while you are making it. You find a tool that is genuinely well built, genuinely popular, and genuinely solves a real problem. You wire it into your system. It doesn’t help. So you wire it in differently. It doesn’t help. So you wire it in a third way, because by now you have invested enough that the question “is this the wrong category of tool” has stopped being available to you.

I spent a day doing that, and the useful part is the number that finally stopped me. It took about sixty seconds to compute and I could have run it on the first morning.

This is a writeup of what we tried, what each attempt measured, and how to tell in advance whether a code knowledge graph will help you. The short version is that a graph is a finding tool, we did not have a finding problem, and there is a cheap way to check which problem you have before you spend a day on it.

The setup

The system is a harness called LLMOS that gives a local 35-billion-parameter model a set of tools for fixing real bugs: reproduce, locate, read a range of lines, patch, run tests, submit. The benchmark is SWE-bench Lite, three hundred real GitHub issues from twelve open source Python projects, where you are graded by whether the project’s own hidden tests go from failing to passing without breaking anything else. The model runs on a single consumer GPU.

The tool is a code knowledge graph. It parses your repository with tree-sitter, deterministically and with no model involved, and turns functions, classes, files, imports, calls and inheritance into nodes and edges. Then you can ask it structural questions. What calls this. What inherits from that. What would be affected if I changed this. It is free to build, it runs locally, it costs no API credits, and the project has ninety-nine thousand stars.

I want to be fair to it throughout, because none of what follows is a criticism of the tool. It does what it says. We pointed it at the wrong thing.

What the tool actually claims

This is the part I skipped, and skipping it is how the day went the way it did.

The project’s headline benchmarks are about conversational memory: retrieval recall and question-answering accuracy on two academic memory datasets, where it does well and does it cheaply. That is not code fixing. The entire code-intelligence claim, in their own benchmark document, is one paragraph. On a roughly one-million-line production repository, giving a coding agent one graph tool lifted key-fact coverage on a graded question set from seventy point eight percent to eighty-two percent, at about a hundred and forty thousand tokens per query. The question set has six questions.

Six. And the task is answering questions about a codebase, not producing a patch that passes a hidden test.

I did not read that paragraph until after the third attempt had already failed. Had I read it first, I would have noticed that nobody was claiming the thing I was hoping for.

Attempt one: let the graph choose which tests to run

When the model patches a source file, the harness runs some nearby existing tests to check it hasn’t broken anything. Choosing which tests count as nearby was done by comparing file paths, which is a guess. The graph knows which tests actually touch the code you changed, so it should choose better.

There was a beautiful motivating case. On one Django instance the model’s fix was one expression away from correct, its own checks all came back green, and it submitted. Asking the graph about the class it had edited returned, immediately, both of the test modules that the official grading would use. Path similarity ranked neither. It looked decisive.

So I measured it properly. Forty-two instances across twelve repositories, both rankers asked for their top six test files, scored on whether the graded test module appeared.

Path similarity found it in sixty-nine percent of cases. The graph found it in thirty-eight percent. The graph found it in cases path similarity missed exactly twice.

Two out of forty-two, and the graph had nothing to say at all in forty-three percent of cases.

I will note one thing in my own favour and one against. In favour: my first version of this measurement said twenty-six percent, and that was my bug, not the tool’s. I had taken only the first hunk of each reference patch, and for many patches the first hunk is the import block at the top of the file, which has no enclosing function to ask about. Fixing that moved the graph from twenty-six to thirty-eight. Against: the number that mattered, the two, did not move at all, and I had already built and shipped the mechanism before running any of this.

Attempt two: fire it automatically instead of offering it

The obvious next thought was that the graph was fine and the model just wasn’t using it. I had exposed it as a tool the model could call, and across eleven instances and several hundred tool calls it called it once. When it did, it asked about a symbol named “format,” which in a Python codebase matches too many things to resolve, and got nothing back.

I concluded that offering a tool is exhortation and firing it is structure, and rebuilt it to fire automatically after every patch, attaching the call graph around whatever had just been edited.

Two things about that conclusion. First, it was right, and I later found two independent practitioners who had arrived at it before me. One of them wrote, plainly, that agents still reach for grep by default even with the graph tools installed and a cheatsheet in front of them, and had resorted to intercepting grep calls and answering them from the graph instead. Second, my stated evidence for it was wrong. I told my collaborator the model had called the tool zero times. It had called it once. I had checked two instances and generalised, which is the same error I have made before and apparently will make again.

The automatic version fired, picked sensible symbols on its own, and produced no measurable improvement. It also arrives at the wrong moment, which took me embarrassingly long to see: it only runs after the model has already found the site and edited it. If you already know where to look, you don’t need a map.

Attempt three: just put the map in the context

My collaborator asked the obvious question I hadn’t. Why all the machinery? Build the graph once, hand the model the map, let it start oriented.

That is what the successful practitioners do, and it is the right instinct. So I generated the compact structural map for Django, the most heavily connected nodes in the repository, the architectural hubs. It is eight hundred bytes and it reads like this: override_settings, SimpleTestCase, TestCase, reverse, GenericForeignKey, cached_property, F, Q, HttpResponse, ValidationError, ModelAdmin.

Those are the most famous identifiers in Django. The top three are test scaffolding, an artifact of the test suite outweighing the source in the graph. Any model that has read the internet knows every one of them.

Which surfaced the thing I now think is the most transferable point in this whole exercise. The value of a map is proportional to the reader’s ignorance of the territory. The practitioner who got the most striking result from this tool ran it on a private React Native application the model had never seen, and the graph surfaced a retired helper still silently loading a heavy security module on every screen boot. SWE-bench is twelve of the most famous open source repositories in Python. The model has effectively memorised them. We are close to the worst possible case for a structural map, and I had not thought about that at all.

The scoped version of the idea survives the objection better, so I measured that too: take the identifiers the issue text actually names, query this exact commit’s graph, see whether the file the fix belongs in turns up. Forty-six instances. The graph found it seventeen percent of the time. Plain grep on the same identifiers found it seventy-eight percent of the time. The graph found it where grep did not exactly once.

Three designs, three measurements, all marginal. Two out of forty-two, one out of forty-six, and a map of things already known.

The number I should have run first

Here is the sixty-second check that explains all three results.

For each instance, compare the file the model actually edited against the file the official fix edits. Not whether the patch was right. Just whether it was in the right place.

Sixteen of eighteen.

The model finds the right file almost every time. It was never lost. Every one of those three mechanisms was a better map for someone who already knew where they were going.

And there is a second number in the same spirit. A knowledge graph’s primary claimed benefit is token economy: you learn what is connected to what without reading the files. The measured win in the field reports is real, around a seventy percent reduction on navigation-heavy work. Our context usage peaks at sixty-two thousand nine hundred tokens against a hundred and thirty-one thousand available. We have sixty-eight thousand tokens going unused every turn. The resource the tool conserves is the one we have a surplus of.

Both of those numbers were available on day one. Neither requires the tool to be installed.

What we actually needed

The failures, when I finally looked at them as failures rather than as opportunities to try the graph again, are all the same shape. The model reaches the right file, makes a change that is entirely plausible, and has no way to find out whether it is right.

One instance differs from the official fix by a single expression: the model returns the value unchanged where the accepted fix returns an empty string. Same file, same method, one token apart. Its own reproduction script asserted a property that its wrong answer satisfied, so it went green and the model submitted.

Another is worse and is entirely our fault. The model produced the correct patch on turn seven. It then worked out that its own test was wrong and wrote four replacements, all of which passed. Our harness refused all four, because of a rule that a script can only become the registered reproduction by failing. So a correct test of an already-fixed bug is ineligible by construction, and there is no path from a wrong reproduction to a right one. After three refusals it told the model to stop writing tests and go change the patch. It did, for five more strategies and three thousand five hundred seconds, before a fallback reached back and used the turn-seven patch, which graded correct all along.

Across seven instances that exhausted every fix strategy we have, not one ever reached a green reproduction, and every single one ended up using the patch from the very first strategy. Roughly six and a half hours of eleven went into work after the answer had already been found.

None of that is a navigation problem. No map fixes it. The fix is to let a passing script prove itself: revert to the original code, run it there, and if it fails on the original and passes with the patch, that is by definition a valid reproduction that this patch resolves. It also makes the guard stricter rather than looser, since both halves must now hold.

How to tell in advance

If you are considering a code knowledge graph for an agent, three questions, in this order, and each is cheaper than installing it.

Does your agent end up in the right file? Compare the files it edits against the files that should have changed. If that number is already high, a finding tool has nothing to give you. Ours was sixteen of eighteen.

Is context actually your constraint? Look at peak token usage against your window. If you are running at half capacity, the thing a graph saves is not scarce for you.

Does the model already know this codebase? A map is worth the most where the reader is most ignorant. Private code, unfamiliar frameworks, a system nobody remembers writing: high value. Famous open source repositories the model has memorised: close to zero.

If all three answers point the wrong way, the tool is fine and you are aiming it at the wrong problem, which is what I did for a day.

What I would tell myself on the first morning

Read what the tool claims before deciding what you want it to do. The claim was one paragraph, on six questions, about answering questions rather than fixing bugs. That was knowable in five minutes.

One beautiful instance is not evidence. Three separate times a single case pointed clearly in a direction, and all three times the version with forty-something instances came back flat. The last one was genuinely striking and still turned out to be one in forty-six. If your motivating example is a single case, the honest next step is the measurement, not the mechanism.

And measure the problem before you buy the solution. Everything above is downstream of never having asked whether finding was hard for us. It wasn’t. It hadn’t been for weeks. The answer was sitting in the run records the whole time, sixty seconds away, and I went looking for it only after someone asked me a question plain enough that I couldn’t route around it.