Everyone Is Writing Skills for Their Agents. Almost Nobody Can Say When They’re Complete.

Simon Willison called Skills “a bigger deal than MCP.” He’s right — but a skill file is only as good as the expert knowledge inside it. Here’s a framework for getting that knowledge out of human heads. I call it expertise distillation.

When Simon Willison wrote that Claude Skills were “awesome, maybe a bigger deal than MCP,” the internet largely agreed — a Karpathy post later, a single SKILL.MD file sits atop a 188k-star GitHub repo, and every team I talk to is busy packaging its know-how into skill folders. Fine. The format question is settled: procedural knowledge lives in files now.

Nobody talks about the harder question. Where does the knowledge inside those files come from — and how do you know when it’s complete?

I build agents for organizations whose real asset isn’t data or process diagrams. It’s the judgment of a handful of senior people — the ones whose expertise effectively is the business. And after a year of this work, I’m convinced the central engineering problem of applied agents isn’t orchestration, isn’t model choice, isn’t the framework du jour. It’s about getting expertise out of human heads and into an agent context, with a way to know when you’re done. I call the discipline expertise distillation — deliberately stealing a word the ML crowd reserves for compressing one model into another. The industry distills models. This is about distilling people.

Why can’t you just ask

The obvious approach fails, and it fails in a specific, predictable way that’s worth understanding before building anything.

You sit down with the expert and ask them to write down their rules. They take it seriously. They produce a sincere, detailed document. And the agent built on that document breaks — not on hard cases, but on things so basic that nobody thought to say them out loud. That’s not the expert being careless. It’s structural: people have no access to the list of things they know. The most important rules are the ones an expert has used ten thousand times and stated exactly never, because they don’t experience them as knowledge — they experience them as the way the world works. Michael Polanyi named this in 1966: *we know more than we can tell.* The expert-systems generation of the 1980s died on precisely this rock and left it a name — the knowledge acquisition bottleneck.

There’s a second structural problem stacked on top: you can’t audit the model either. You don’t know what the LLM absorbed about the client’s domain in pretraining, so you can’t know in advance which “obvious” things need saying. The only safe assumption is that it knows nothing about this specific task.

Put those together, and the conclusion writes itself. No interview can be complete, and no one can verify completeness by reading the document. Completeness has to be measured. That measurement is the core of the framework.

The distillation loop

The framework rests on one shift of perspective: stop collecting what experts say about their work, and start collecting the work itself. Then use the work to interrogate the context.

It runs as a loop with five stations.

Ground truth. Collect real expert outputs on real cases — decisions, emails, approvals, escalation resolutions. Not descriptions of how they’d decide; actual decisions, with their full texture. This is the raw material everything else feeds on.

Decomposition. Break each expert answer into atomic claims — small, individually checkable statements. This step looks mechanical and isn’t: claims that are too fine-grained lose the context needed to verify them, and trivial claims inflate your scores. The working rule: each claim should carry just enough context to be checkable on its own, and normative judgment calls that nothing could “support” should be filtered out rather than counted as gaps.

Verification. A second agent takes each claim and asks one brutal question: could the agent’s current context — its rules, documents, and tools — have produced this? Every claim that fails is treasure: a piece of the expert’s head that never made it onto paper, caught red-handed. Note the direction of the check. Most evaluations ask, “Is the agent’s output supported?” This asks the inverse: “is the expert’s output supported by what the agent has?” That inversion is what turns evaluation into gap detection.

Routing. An unsupported claim means one of two very different things, and confusing them wastes everyone’s time. Sometimes it’s a missing rule — the expert knew something the context didn’t say, and a human needs to write it down. But surprisingly often it’s a missing tool — the expert’s answer relied on looking something up, checking a system, or running a calculation. No paragraph of prose will substitute for an API the agent doesn’t have. So every gap gets triaged: rules go to the expert, tools go to the engineers. In practice, this routing decision is where most of the judgment lies, and it’s the step I’ve never seen described elsewhere.

Convergence. Both streams feed back into the context, and the loop runs again on the next batch of cases. The number to watch is the claim-support rate — the fraction of expert claims the context can now account for. It climbs, week over week, and when it flattens, you have something no workshop or interview marathon can promise: a context whose completeness you measured instead of assumed.

In effect, this is training on answers — just not through gradient descent. Through text. The learned artifact isn’t a weight matrix; it’s a playbook the client’s own experts can read, correct, and own.

How to ask, once the loop fires

The loop tells you where the gaps are. Getting the expert to fill them is its own craft, and here the useful literature is older than most of the people deploying agents.

Can we ask an expert, “What’s the rule?” Under direct questioning, experts recite a rehearsed self-portrait — the tidy dinner-party version of their job. The knowledge-engineering field established this decades ago, along with what actually works: contrast. Bring two cases — the one where the agent failed, next to a nearly identical one where it succeeded — and ask a single question: what’s different? Presented with a contrast, the expert doesn’t recite; they notice. The discriminating rule they articulate is routinely one they would never have produced from “please list your rules.” The distillation loop feeds this technique perfectly: every unsupported claim arrives with its case attached, ready to be laid next to its twin.

A warning about the machinery itself: don’t expect agents to do this probing unprompted. Models demonstrably notice ambiguity — it shows up in their reasoning traces — and then barrel ahead anyway. Curiosity has to be engineered in as an explicit policy, not hoped for as an emergent property.

Three channels, one auditor

Expertise doesn’t arrive through one pipe, and the channels disagree — the disagreements are where the gold is.

What experts say (interviews) captures judgment and exceptions. What experts do (demonstrations, screen recordings, work traces) captures the say/do gap — procedures people follow without knowing they follow them, including the unofficial shortcuts that are sometimes genuinely better than the documented process. What experts correct (production escalations, edits to agent output) captures everything nobody thought about until it broke. A mature distillation practice runs all three intakes and treats disagreement between channels as a signal rather than noise.

The claim-verification loop sits above all three as the audit layer. It doesn’t care which channel a piece of knowledge came through; it only asks whether the expert’s actual outputs are now derivable from the accumulated context. That’s what makes it a completeness check rather than just another intake.

Two operational scars worth passing on. First: never let anything — optimizer, cleanup pass, well-meaning colleague — rewrite the playbook wholesale. Iterative rewriting quietly dissolves hard-won edge cases; everything should be small, reviewable deltas, with a human on the merge button. Second: resist the urge to make the playbook pretty. The compression instinct is the enemy — the ugly, specific detail is the value. (The Stanford researchers behind Agentic Context Engineering recently gave these failure modes proper names — context collapse and brevity bias — and reading their paper felt like finding my incident reports formatted in LaTeX.)

The field is proving the premise

If “learning through text instead of gradients” sounds like a consultant’s cope, the research of the last eighteen months says otherwise.

The technique of having one model read another’s failures and rewrite its instructions was developed for a 2025 *Nature paper—the most comprehensive in terms of techniques. This year at ICLR, a system called GEPA showed that evolving prompts through natural-language reflection beats reinforcement-learning fine-tuning while burning 35 times less compute. Editing text outperformed editing weights. The intuition is the same one that powers distillation: a number tells a system that it failed; a sentence tells it why*. Language is a denser learning signal than a gradient.

Yet when I went looking for the distillation loop itself — expert ground truth, decomposed into claims, verified against context, iterated to completeness — I found only its parts. The verification machinery is standard science now; the RAG-evaluation world even ships the core metric under names like claim recall. What nobody has published, and nobody sells, is the assembled thing: gap detection as a completion driver, running until an expert’s actual wisdom is provably in the files. The routing step — is this a gap for a human or a tool for an engineer? — appears nowhere at all. One evaluation framework’s docs admit the blind spot with disarming honesty: they verify your agent is hallucination-free “up to the limit of its knowledge base.” Is the knowledge base complete? Out of scope. For everyone.

Every LLMOps platform now ships a prompt-optimization loop; that part is a commodity. The completeness loop is still open country.

The part that compounds

Models get swapped. Orchestration frameworks get replaced. Over a year of engagements, every layer of the stack has turned over except one: the distilled expertise. Playbooks and skill files move across model migrations untouched, their claim-support metrics moving with them. Text, it turns out, is the most durable layer in the whole architecture — and the only one the client’s experts can read, audit, and genuinely own.

So the framework, compressed to a paragraph: stop treating context as something you write, and start treating it as something you converge. Assume the model knows nothing about the domain. Assume the expert can’t tell you everything — not won’t, can’t. Put a measurement between them and let it drive: real answers, decomposed claim by claim, verified against the files, gaps routed to humans or engineers, support rate climbing until it flattens. Small deltas only. Human on the merge button.

Willison was right that Skills are a bigger deal than MCP. But the bigger deal still is what fills them. Distilling models into smaller models is a solved problem. Distilling people into context — one caught claim, one recovered sentence at a time — is the craft the next generation of agents will actually be built on.

Selected reading, for those who want the receipts: GEPA (arXiv:2507.19457, ICLR 2026) · TextGrad (Nature, 2025) · Agentic Context Engineering (arXiv:2510.04618, ICLR 2026) · FactScore (arXiv:2305.14251) · RAGChecker (arXiv:2408.08067) · GATE (arXiv:2310.11589) · Shadbolt & Smart, “Knowledge Elicitation” (2015) · Simon Willison, “Claude Skills are awesome, maybe a bigger deal than MCP” (2025) · Anthropic, “Equipping Agents for the Real World with Agent Skills” (2025).