From Agent Memory to Portable Skills

Grounded skill distillation on a context graph in the Neo4j Agent Memory Service (NAMS)

An agent accumulates hard-won procedural knowledge inside a single workspace and single context window: how to triage a P1 incident, which tool to call first, or what “done” looks like. This learned experience captures how your organization actually gets things done, but open a fresh session and the agent starts cold.

NAMS (Neo4j Agent Memory Service) gives agents persistent, graph-based memory, so a new session can look up what happened. But recall is not capability: knowing how last month’s P1 triage went does not hand the next agent the procedure to triage the next incident. This post is about the transformation that closes that gap: skill distillation, turning a scoped slice of a workspace’s memory graph into a portable, provenance-grounded Agent Skill (a SKILL.md package) an agent can load and run.

Skill distillation produces knowledge your agents can act on, an artifact they load at runtime, and a governance loop that keeps both honest.

Agents work, memory grows, a scope is distilled into a skill, and the next agent loads it. Governance re-distills when the memory underneath moves.

From Information to Knowledge to Action

A NAMS workspace holds three memory types which together make up a context graph of your organization, exposed through agent interactions.

• Short-term messages and conversations
• Long-term memory knowledge graph of entities and their relationships, extracted using a domain-specific ontology, mapping to your organization’s data model
• Reasoning memory that captures agent reasoning and decision traces, and agent execution paths (tool calls, tool results, reasoning traces, and feedback)

Short-term memory is information: a record of what was said and done that you still have to re-read to learn from. Long-term memory is knowledge, because those same conversations are extracted through a domain-specific ontology. The ontology is the load-bearing part: it decides that a conversation contains an incident and the service it touched, not a paragraph that mentions both. That is the difference between text you can search and knowledge you can query.

Action is the rung memory alone never reaches: knowledge you can query still has to be found and re-assembled into a plan every session. Reasoning memory puts it in reach by recording the reasoning, the action, and the outcome of every step an agent took. Distillation turns the recurring shape of that work into a procedure a fresh agent executes rather than re-derives.

Memory records what happened. An ontology turns what happened into knowledge. Distillation turns that knowledge into something an agent can run.

A domain ontology turns conversation into knowledge. Skill distillation turns knowledge into something an agent can run.

Distilling a Skill from Memory

Skill distillation in NAMS can be triggered via the HTTP API, from an MCP client, or from the dashboard UI. The scope determines everything the distiller is allowed to look at — and because every claim has to cite something inside it, everything the skill package is allowed to include.

The skill distillation and governance feature in the Neo4j Agent Memory Service (NAMS) distills actionable knowledge from agent memory stored in Neo4j.

What Distillation Produces

What comes back from the skill distillation process is a spec-compliant SKILL.md package. Frontmatter contains provenance metadata, the body leads with the procedure, and reference material is included.

What separates it from a hand-written skill is the evidence shipped next to the procedure: steps grounded in actual tool calls observed from reasoning memory, grounding and coverage scores, and claims linked to the sources that justify them. The skill persists as a subgraph in Neo4j with GROUNDED_IN edges pointing back into the memory that justifies each claim.

Every claim carries its grounding: the memory entities and decisions behind it, a grounding score, and the exact SKILL.md that was distilled.

Nothing Ships Unreviewed

A distilled skill lands pending human review. A reviewer reads the steps, follows the evidence back to the work behind them, and approves or rejects.

Each step in the review UI shows its tool binding, its input and output contract, the condition that marks it done, and the memory that supports it.

Keeping Skills Honest: Governance and Drift

A grounded skill is only as good as the memory under it and memory changes over time. Entities merge, policies are updated, and tool behavior shifts. Because a skill and its grounding are graph data side by side, “is this skill still valid?” is a graph query.

Contradiction drift — when the world changes, not the citations. The skill-drift check in NAMS compares a step’s recorded expectStatus against the current status of the :ToolCall nodes it is grounded in. A step that used to succeed but whose tool now fails is flagged as contradicted, even though every grounding edge is still present.

The NAMS skill library and review UI shows skills in the process of being reviewed and flags any governance related issues such as drift from the underlying memory graph.

Bounded, Node-Level Repair

When a single step drifts, you don’t re-derive the skill. A repairStep takes the workspace lock, re-grounds only that step by following merged sources to their survivors (and dropping deleted ones), re-validates the graph, and cuts a patch version that supersedes the prior — every other step untouched, so a reviewer sees a one-step diff.

If no surviving grounding exists, repair declines and flags a full re-distill — it fixes salvageable drift, never papers over the unsalvageable.

Published skills are re-checked against live memory. When one step drifts, only that step is repaired; the rest of the skill is untouched.

How a Published Skill Reaches Your Agents

Because the artifact is a standard SKILL.md package, a published skill can be loaded into Claude Code as a skill, fetched by MCP clients through the NAMS MCP server, or consumed over REST by any other agent framework. The agent picking it up only has to know how to read a skill.

That closes the loop: agents work, memory grows, a scope is distilled, and the next agent loads the result.

Because the artifact is a standard SKILL.md package, the same skill loads into Claude Code, MCP clients, REST callers, or any other agent framework.

Under the Hood: Why Prose Skills Break

The skill distillation functionality in NAMS is built using the Agent Instruction Protocol (AIP) for representing and governing agent skills.

The “skill” most agent systems ship is a Markdown file of prose instructions. The prose format flattens away the decisions, hierarchies, and dependencies inherent in the skill’s structure — none of them explicit, navigable, or composable.

That format has several compounding problems, which the AIP research names crisply:

• Re-derivation cost. The agent linearly re-reads and re-interprets the prose every session — slow and token-heavy, with run-to-run variance.
• Prompt brittleness. Small, semantics-preserving wording changes can swing task accuracy by tens of points.
• Resistance to improvement. Agents struggle to revise domain prose they don’t fully understand and exhibit an additive bias — they pile on words rather than fix structure.
• Opaque failure. There are no addressable units, so you can’t attribute a failure to a specific step.

[embed]AIP: A Graph Representation for Learning and Governing Agent Skills Agent Skills today consist largely of free-form prose requiring the agent to read, interpret, and re-derive how to act…arxiv.org

What AIP Proposes

AIP’s answer is to represent a structured skill as a typed, schema-validated graph: a skill node (purpose, triggers, scope) over step nodes that are either script-backed (deterministic computation) or description-based (judgment), connected by typed I/O and dependency relationships, with a validation gate at authoring time and node-level addressability for diagnosis and repair.

The AIP paper frames this as a validity-gated action space: because each node is typed and schema-bound, edits are constrained and checkable — a foundation for structured, measurable skill improvement instead of open-ended prose rewriting.

This graph structure measurably helps. In the AIP authors’ own evaluation on SkillsBench, structured skills beat freeform prose by +14.1 percentage points in pass rate (53.3% → 67.4%) and ran ~13% faster (585s → 510s). (Source: AIP, arXiv 2606.04781)

The same procedure as prose and as a typed graph. On SkillsBench, the graph form beat prose by 14.1 points in pass rate and ran about 13% faster. (Source: AIP, arXiv 2606.04781)

AIP Procedure Graphs in NAMS

NAMS is a hosted memory service from Neo4j Labs, well-positioned to adopt the skill-as-graph approach pioneered by AIP: its memory model already records the reasoning, action, and outcome substrate AIP steps need, and skills live in the same graph database as the memory that grounds them.

So a distilled skill is a typed, schema-validated execution graph, falling back to prose only when the scoped memory is too thin to support one. The topology is derived deterministically from the traces, not written by a model. A step is script-backed exactly when a recorded tool call grounds it — the tool name, typed inputs and outputs, and expected status all come from the recorded call — while judgment steps stay description-based, grounded in the agent’s recorded reasoning. Dependencies and branch types (sequence, one-of, parallel, retry) are inferred from how steps actually followed each other in each conversation, and a scope with fewer than three recorded steps degrades gracefully to prose instead of shipping a skeletal graph.

Inside the Distillation Pipeline

Distillation is a graph-backed, pollable job. A trigger enqueues a run on a Redis stream; the distiller runs a seven-stage pipeline:

1. Scope — resolve the requested scope to the set of conversations the skill will draw from
2. Snapshot — take a per-workspace lock and wait until scoped memory has settled, so a skill is never distilled from half-extracted memory
3. Consolidate — group recurring actions into frequency-weighted pattern clusters, splitting effective patterns from anti-patterns by tool-call outcome
4. Graph — derive the procedure-step topology deterministically from the recorded traces
5. Synthesize — the one LLM stage: write the skill’s claims and step annotations, every one citing source ids
6. Gates — grounding, coverage, coherence, spec lint, and PII checks; a scope that fails is withheld, not shipped
7. Persist — write the skill subgraph and the packaged artifact into Neo4j, pending human review

Its raw inputs come from all three memory types: recorded decisions (:AgentStep nodes with reasoning, action, and result), the tool calls each decision made (:ToolCall nodes with tool name, input, output, and recorded status), the entities those conversations touched, and the surrounding messages, observations, and reflections.

For synthesis, the distiller calls out to a Python grounded-synthesis endpoint, and the LLM’s role there is deliberately narrow. It sees up to 400 interleaved, PII-redacted source snippets and returns the skill’s claims and step annotations (names, descriptions, completion criteria), each carrying source references. Claims without sources are dropped before they leave the synthesis service, and the worker re-checks every reference against the actual subgraph: the graph, not the model, is the authority on what exists. The LLM never invents structure, sources, or code.

The gates are what keep a bad scope from becoming a bad skill. The grounding score must clear 0.9 and coverage of recurring patterns must clear 0.6; a scope that contains two or more separable procedures is withheld with a split suggestion rather than shipped as one blended skill; exemplars are PII-redacted; and the procedure graph is schema-validated before anything persists.

Distillation runs six deterministic stages around a single LLM call, with gates that can withhold a skill before anything is written back.

Try It Today

Skill distillation and governance are live in NAMS now. NAMS is a Neo4j Labs project, and we’d love your feedback — tell us what works and what you’d like to see next.

Areas we’re exploring include deeper procedure-graph editing, agent-driven skill selection, and additional framework integrations.

[embed]Going Meta S03E10: a live walkthrough of skill distillation in NAMS.

Resources

• Try it now in NAMS: memory.neo4jlabs.com
• Read the docs
• NAMS Skills distillation livestream on Going Meta
• NAMS walkthrough blog post