The Evolution of TokenSave

Saving tokens, and then a great deal more.

TokenSave shipped its 1.0.0 release on March 24, 2026. It could index Rust and a dozen other languages, serve a handful of MCP tools over stdio, and stop Claude Code from burning tokens re-reading the same files. Useful, but narrow: a read-only lens onto a codebase.

Roughly ten weeks and more than a hundred releases later, version 6.1.2 is a different kind of program. It parses more than forty languages, scores the structural health of a codebase on a single tamper-resistant number, edits files by symbol name without shelling out to a regex, runs the compiler and maps every error back to the graph, runs the affected tests, and remembers decisions across sessions. It still ships as one native binary with no runtime dependencies.

This is the story of how it got there. Not a changelog read end to end, but the shape of the decisions: the problems that surfaced, the features they demanded, and the things that were built with conviction and later torn back out.

The trajectory at a glance

The language explosion

Version 1.0.0 covered the languages most professional teams actually use: Rust, TypeScript, Python, C, C++, Kotlin, Dart, C#, Pascal, plus Java, Go, and Scala carried over from the pre-1.0 days. That handled “most” codebases, and the requests for the rest arrived immediately.

PHP and Ruby came in 1.4.2. Then 2.0.0 doubled the count in a single release, adding sixteen extractors and reaching thirty languages. The list reads like a tour through computing history: Swift and Zig for the modern crowd, Bash and Perl for the Unix faithful, Protobuf for the microservices world, Nix for the reproducible-build enthusiasts, and the deep cuts beyond, Fortran and COBOL and Objective-C and three flavors of BASIC. QuickBASIC 4.5 followed in 2.1.0 with its own file extensions.

The work did not stop at thirty-one. GLSL shader files landed in 4.0.6. A markdown extractor in 4.1.8 turned headings into a navigable structure. The biggest single jump after 2.0.0 came in 4.3.2, which added nine languages aimed squarely at the functional and data-science worlds that the earlier waves had skipped: R, SQL, Julia, Haskell, OCaml, Clojure, Erlang, Elixir, and F#. Most recently, 6.1.2 added Svelte and Astro, both of which work by isolating the embedded TypeScript and delegating to the existing extractor rather than inventing a parallel parser.

Each extractor does real semantic work rather than wrapping grep. The Nix extractor resolves imports into cross-file dependency edges and reads derivation fields out of mkDerivation calls. The Objective-C extractor follows inheritance through protocol conformance. The Erlang extractor records arity-qualified names like foo/2. None of this is a list of regular expressions.

That many grammars created a binary-size problem, because not everyone needs to parse COBOL. Version 2.0.0 answered with feature-flag tiers: a lite build compiles the core languages, medium adds more, and the default full includes everything. Individual lang-* flags let a user cherry-pick exactly what they want and nothing else.

Four waves of tools, then a fifth that changed the premise

The MCP tool surface grew past seventy entries, and it grew in distinct waves, each answering a class of question that agents kept asking.

The first wave was navigation: search, context, callers, callees, node lookup, the files and affected-tests queries. These replaced the bulk of what an Explore agent does with grep and glob.

The second wave was structural analysis: dead-code detection, circular-dependency finding, module API surfaces, unused imports, semantic changelogs between git refs, and rename previews. Each was a graph query rather than a file-by-file scan.

The third wave was quality and compliance: complexity ranking computed from the AST during indexing, recursion detection, god-class detection, documentation coverage, and the porting tools for teams migrating a codebase across languages.

The fourth wave was workflow integration and multi-branch awareness: semantic summaries of uncommitted changes for commit messages, semantic diffs for pull-request descriptions, symbol-level test mapping, and the branch tools that let an agent search and diff another branch without switching the checkout.

For its first four waves, every one of those tools was read-only. TokenSave looked at code and answered questions about it. The fifth wave broke that premise, and it is worth its own section.

The writing tools

The turning point arrived quietly in 4.1.5, contributed by @pierreaubert, with four edit primitives: a unique-match string replace that refuses to act if the target appears zero times or more than once, an atomic multi-replace that applies every change or none, an insert-at-anchor tool, and a structural rewrite that drives ast-grep. Each one writes the file and then re-indexes it, so the graph never drifts out of sync with the bytes on disk. Suddenly TokenSave was not only a way to read a codebase, it was a safe way to change one without the regex-and-shell-quoting hazards that make blind edits dangerous.

The early versions had the rough edges you would expect from a tool that mutates files. Insert-at quietly stripped the trailing newline from a file until 4.3.5 re-appended it. Edits to file types with no registered extractor reported failure even though the write had already committed, fixed in 4.3.1 by letting the re-index step succeed as a no-op. Multi-byte UTF-8 in a failure preview could panic until 5.1.2 routed everything through a shared, byte-budgeted prefix helper.

The real maturation came in 6.1.0, when editing became symbol-aware. Two new tools, a symbol replace and a symbol insert, take a name rather than a literal string. They resolve it by exact qualified-name match, narrow to callable kinds when a name is ambiguous, and refuse the edit outright rather than guess if more than one callable still matches. Instead of “find this exact text and swap it,” an agent can now say “replace the body of this function” and trust that it will land on the right one or fail loudly. This is the same capability that competing tools expose, built here on top of the graph that TokenSave already maintains.

Health-based scoring

The second large theme since 4.0 is the idea that a code graph is not only a map for navigation, it is a substrate for measuring structural quality. Version 4.2.0 introduced a suite of health and structural-analysis tools, and at its center sits a single composite number called the quality_signal.

The signal runs from zero to ten thousand and is computed from independent dimensions of structural health. The original five were acyclicity, dependency depth, equality of complexity distribution, functional redundancy, and modularity. The crucial design choice is the aggregation: the dimensions are combined with a geometric mean rather than a sum or an average. A geometric mean collapses toward zero if any single factor is bad, so a codebase cannot paper over a tangle of circular dependencies by being tidy everywhere else. No one dimension can be gamed in isolation.

Around that headline number sits a family of supporting tools. A Gini coefficient identifies god files and uneven complexity distribution. A dependency-depth tool reconstructs the longest file-level chains using Lakos levelization, surfacing the kind of transitive fragility that direct coupling metrics miss. A Design Structure Matrix exposes hidden coupling and layering violations. A risk-weighted test-gap tool blends complexity, fan-in, coverage, and ninety days of git churn into a single “where should the next test go” score. Two session tools snapshot the health metrics before an AI coding session and diff against them afterward, reporting per-dimension deltas so you can see whether an agent’s work left the structure better or worse.

This is also where the scoring becomes more than a report. A single tamper-resistant number that moves up when structure improves and down when it degrades is exactly the shape of objective that Claude Code’s new /goal feature wants: a measurable target the agent can steer toward rather than a vague instruction to “clean things up.” Point /goal at the quality_signal, let the session tools capture a baseline before work begins, and the agent has a concrete gradient to climb and an honest scorecard at the end. Because the signal is a geometric mean, the goal cannot be satisfied by cosmetic wins that leave a real structural problem untouched, which is precisely the property you want in a target an autonomous agent is optimizing against.

The scoring kept getting refined. A coverage-discipline dimension was added so that genuinely untestable functions can be annotated with a doc-comment convention rather than dragging the score down, and when the early penalty turned out to punish honest annotation, 4.5.0 reduced it. Version 6.0.0 added a functional-redundancy tool of its own that fingerprints every function body four ways, by AST shape, control-flow graph, ordered call sequence, and a token-shingle set, and blends them into a similarity score that flags real copy-paste duplication across the codebase. The same release added a details mode that breaks the composite signal back down into its per-dimension parts with raw counts and plain-language interpretations, so the headline number is auditable rather than opaque.

Closing the loop: diagnose, fix, test

A read-only graph can tell you what calls what. It cannot tell you whether the code compiles. Version 4.8.0 closed that gap with two tools that, together, turn TokenSave into a small build-and-test harness. The first runs the compiler or type checker and parses the raw output into structured diagnostics, then maps each error back to the smallest graph node that contains it and pre-attaches the callers the broken code is reachable from. The second walks the graph to find every test covering the changed files and runs exactly those, with a timeout and a cap so a sweeping refactor cannot dispatch an unbounded list. Version 5.0.0 generalized the diagnostics tool across cargo, tsc, and pyright, and forced the compiler’s target directory into a private path so it can never race with the developer’s own interactive builds.

The pattern here is consistent: take a loop that an agent used to run by hand, shell out, parse text, read a file, follow a reference, and collapse it into a single structured response.

Memory and the savings ledger

Version 4.5.0 gave agents a memory. Three tools let an agent record a decision, mark a code area it worked on, and recall both later through a fuzzy full-text search, all persisted in the per-project database so they survive across sessions. The same release made the project’s reason for existing measurable. Every tool call now writes an append-only row to a savings ledger, and a gain command reports tokens saved and a dollar estimate based on current model pricing refreshed daily. A companion benchmark runs a fixed query set and reports retrieval savings against a full-file baseline; on TokenSave’s own repository it measured a ninety-three percent mean saving across ten generic queries.

Version 5.0.0 added a cross-session response cache underneath the new read tool. A re-read of an unchanged file returns a roughly thirty-token “unchanged” stub instead of the whole file again, and the cache key folds in the last sync time so a forced re-index correctly invalidates it.

The correctness reckoning on real repositories

Somewhere around the 4.10 to 4.14 series, the project turned its attention from breadth to truth. Running the tools against large real-world codebases, sonium, scirs, polkadot-sdk, and eventually a full chromium checkout, surfaced a class of bugs that small test fixtures never would.

Most of them traced back to a single root cause: the resolver was fuzzy-binding names. An impl Default for X would bind to whatever local node happened to be named Default, and in one codebase a hundred and fifty such blocks all bound to a single enum variant, swamping the ranking tool with junk. The fix in 4.13.0 was a kind-compatibility matrix that constrains what each edge type is allowed to point at: an implements edge must target a trait or interface, a call must target something callable, an annotation must target a decorator. The same release rewrote cycle detection around an iterative Tarjan strongly-connected-components algorithm so that the circular-dependency and port-order tools report one entry per genuine cycle instead of one per DFS path through it.

This period also produced the project’s most instructive abandoned experiment. The dead-code query was slow because it re-ran a leading-wildcard text match for every candidate row. Version 4.14.8 tried to fix it by lifting the match into a common table expression, on the theory that the database would evaluate it once. In practice the database did not materialize the single-reference CTE, and on a 76,000-row codebase the query went from a tenth of a second to over a minute. Version 4.14.9 reverted the change and left a “do not lift this into a CTE” comment at the call site. A later attempt that used a single temporary table also failed on chromium, where the optimizer iterated thirteen thousand markers as the outer loop. Only the three-step approach in 5.1.1, resolving markers once, pre-joining into two indexed temporary tables, and probing through them, brought the chromium query from a timeout down to about a second. The inline comment now documents all three failures so the next person does not repeat them.

A parallel set of fixes addressed performance traps in the extractors themselves. Tree-sitter’s child-access is linear in the child index, so an index loop over a node’s children is quadratic. On a twenty-thousand-line C file with monster switch statements, this made indexing appear to hang. The fix replaced every such loop with cursor-based traversal, and the same pattern was hunted down across the Batch, PowerShell, Clojure, and COBOL extractors in a single sweep.

Subtraction as a feature

The most honest part of this history is the list of things that were built well, shipped, and then deliberately removed.

Embeddings went first. Through the 3.x line, TokenSave embedded symbols during indexing and matched them by cosine similarity. Version 4.0.0 deleted the entire vector module and replaced it with an agent-supplied keywords parameter: instead of paying thirty seconds per thousand nodes to build embeddings and carrying a fifty-megabyte model, the calling agent, which is itself a language model, simply provides the synonyms. The trade-off is real, since embeddings can match concepts with no shared words, but the practical wins of zero indexing cost and sub-millisecond lookups carried the decision.

The graph visualizer had an even shorter life. It arrived in 4.0.0 as an interactive browser view with a right-click context menu, and it was gone by 4.0.2, removed in the same period that the upstream sibling project dropped its own.

The largest reversal was the daemon. Version 2.4.0 introduced a real background service that watched every tracked project and synced on change, with platform-specific autostart on macOS, Linux, and Windows. It was substantial enough that its service-management layer was spun out into a separate published crate. Version 3.3.1 taught it to detect its own upgrade and exit cleanly so a stale binary could not keep serving old code. And then version 6.0.0 deleted the whole thing, around eleven hundred lines of launchd plists, systemd units, Windows service registration, PID files, and UAC elevation. File watching moved inside the MCP server itself, which only needs to watch for the duration of an agent session.

That replacement did not survive either. The in-process watcher, even after being tuned to avoid recursively watching ignored directories, was still the source of severe memory pressure on large monorepos, with one user reporting the process climbing toward nineteen gigabytes before the kernel killed it. Version 6.1.1 removed the watcher entirely and replaced it with a lazy staleness check: a gitignore-aware walk gated by a thirty-second cooldown, run at the top of each tool call. The unbounded-memory class of bug is now structurally impossible rather than merely mitigated.

The lesson across all four removals is the same. A feature that looks like an asset on the feature list can be a liability in the field, and the willingness to delete it is what kept the binary small and the failure modes bounded.

Resilience: never let one file take down a sync

Parsing arbitrary source with C-based grammars means occasionally meeting a file that makes a grammar misbehave. A vendored markdown scanner contained assertions that called abort and core-dumped the entire sync process on certain inputs. Version 4.2.1 added two layers of defense, disabling C assertions in release builds and wrapping every extractor call in a panic guard. Version 4.3.0 went further and moved extraction into short-lived worker subprocesses authenticated by a per-spawn token, so a grammar that segfaults or calls abort, a path Rust cannot intercept, takes down only its worker. The pool respawns it, the file is logged and skipped, and the sync continues. Version 4.3.13 added a per-file timeout backed by a watchdog, so even an infinite-loop grammar pathology, the kind that a GLR markdown parser can hit on certain YAML frontmatter, can no longer hang a sync forever.

The agent ecosystem and the configs underneath it

TokenSave began as a Claude Code plugin. It now installs itself into thirteen different AI coding agents, each with its own configuration format and prompt-file location. The trait-based integration abstraction introduced in 1.8.0 made that tractable, and the roster grew steadily: OpenCode and Codex, Gemini, then Copilot, Cursor, Zed, Cline, and Roo Code, then Antigravity, Kilo, Mistral Vibe, and most recently Kiro.

Writing to a dozen different config files, some of them JSON, some TOML, some JSON-with-comments, is exactly where data loss happens. A bug in 4.3.15 showed how: a single mis-parsed TOML document silently erased every other key in a user’s Codex config. The response was twofold. Every config write across every integration now leaves a backup copy first, writes to a sibling file, and renames it atomically into place, so a crash mid-write cannot leave a half-written config. And the doctor command grew from a simple health check into a self-repair tool that validates hook shapes, detects legacy formats, fixes broken subcommands in place, and purges stale entries from the global database.

Benchmarks and good neighbors

Recent releases put numbers behind the claims. A benchmark harness drives TokenSave and a competing tool, token-savior, side by side on the same FastAPI clone with a shared random query sample, and reports cold-index indexing roughly three times faster and impact analysis dozens of times faster. A fork of that competitor’s own ninety-six-task agent benchmark scored 184 of 192 on a first untuned attempt. A full comparison document lays out where each tool is genuinely stronger, including an honest “when to use which” section.

The most interesting recent dynamic is cross-pollination with a sibling project, codegraph. Several of the most valuable 6.x fixes, borrowed-worktree detection, catch-up sync on connect, the focused per-file staleness banner that replaced an all-or-nothing warning, are ported from it by issue number. The two projects now trade improvements rather than diverge.

What the numbers say

The arc from 1.0.0 to 6.1.2 is dense. Languages went from fifteen to more than forty. MCP tools went from a handful to more than seventy. Supported agents went from one to thirteen. Indexing got between thirteen and twenty-six times faster through parallel extraction and bulk-load mode, taking a twenty-eight-thousand-file monorepo from over nine minutes to roughly twenty seconds. The database schema migrated through ten versions, each migration triggering an automatic re-index so a user never has to think about it. The test suite grew past a thousand tests, and the codebase refactored itself more than once along the way, including a pass that decomposed oversized functions to satisfy a strict per-function line limit.

The single largest functional jump after 4.0 was 5.0.0, which added nine tools in one release, a mode-aware file reader, a flat file outline, trait-implementation lookup, an unsafe-pattern scanner, structured diagnostics, a config-file query tool, signature-shape search, a struct-constructor site finder, and a field read-write partitioner, alongside the cross-session read cache and a schema change that folded containment relationships off the edges table and onto the nodes themselves for faster member lookups.

Built in the open

None of this happened in isolation. TokenSave is open source, and a good deal of its best work arrived as pull requests from people who had no obligation to send them. The entire family of edit tools, the primitives that turned a read-only index into something that can safely change code, came from @pierreaubert, who also hunted down the N+1 query patterns in graph traversal and collapsed them from a database round-trip per node into a single batched query. @lesbass made self-upgrade behave on Homebrew installs and taught the indexer to follow symlinked source directories. @LucioPg fixed branch detection inside git worktrees, default-branch resolution, and a clutch of Windows runtime panics. @davidefossacecchi corrected the OpenCode rules file path. Between them they kept the integrations honest across platforms most maintainers never test on.

Just as valuable were the people who filed the bug reports that small fixtures could never have surfaced. The nineteen-gigabyte watcher blowup was diagnosed from a sample trace and an FSEvents-sandbox reproducer that @ottob put together, leaving no ambiguity about the root cause. @AGiorgetti, @uwe-sure, and @xaerogonzo each reported failures, duplicate indexing under mixed path separators, a stale-sync warning that never cleared, that turned into regression tests and stayed fixed. A project that runs across thirteen agents and three operating systems cannot be tested by one person on one machine. The community is what makes the breadth credible, and every contributor who believed enough to open an issue or a pull request is part of why the binary works where it does.

What comes next

The competitive landscape stays interesting. Other tools approach the same problem from different angles, as a prompt-prefill layer, as a multi-repo registry with accuracy benchmarks, or as a set of lifecycle hooks that block redundant reads. Each has an idea worth borrowing, and the porting notes in the recent changelog show that borrowing already happening in both directions.

The language coverage is broad but not total. The feature-flag system makes adding the remaining gaps straightforward without bloating the default binary. The agent ecosystem keeps expanding, and the real cost there is not adding a thirteenth or fourteenth integration but keeping all of them tested across three operating systems.

TokenSave started as a way to make one AI agent stop reading the same files over and over. It has become a code-graph workbench that any agent can query, that scores the structural health of a codebase, edits files by symbol name, runs the compiler and the affected tests, and remembers what it decided last session, all from a single native binary that maintains an honest ledger of the tokens it saves. The core insight has not changed since 1.0. Give the agent a graph instead of making it grep, and let everything else follow from that.

Learn more at tokensave.dev.

Want more like this? I write regularly about Rust, design patterns, and performance. Follow me here on Medium to stay updated.