How I scaled my Obsidian in LISP

What if we use LISP’s homoiconic data structure as the canonical truth layer for Karpathy’s wiki?

A few weeks ago, Andrej Karpathy shared his personal knowledge vault — a lightweight wiki of Markdown files. I watched the wave of inspired implementations follow: LLM + Obsidian, knowledge graphs, cross-platform tools, countless takes on the concept. But when I examined what these systems shared, I noticed two constraints that kept them from being more than reading tools:

1. Scale limits. As Karpathy noted, even a well-curated vault hits a ceiling: ~100 documents, ~400k words.
2. Lack of structure. Flat Markdown doesn’t support operational work — no way to query relationships, enforce consistency, or answer “what changed?”

These two problems were worth solving. The immediate candidate was LISP’s symbolic data structure — something I’d studied years ago in Paul Graham’s work. I asked two AI models whether S-expressions could push back against these constraints. Both said yes. So I built a LISP-wiki that uses S-expressions as the canonical truth layer.

Why homoiconicity matters

LISP’s defining elegance is simple: code and data are indistinguishable. This property, called homoiconicity, means a program can treat its own structure as data — inspect it, modify it, even generate new code on the fly.

For a knowledge vault, this changes everything. Instead of a database schema that requires migrations, or a semantic layer that requires an ORM (Object-Relational Mapping — the plumbing that translates between code objects and database tables), I get a single, uniform data structure. Every node is a Lisp plist (a simple key-value structure). Relations are first-class data. The entire graph can be queried with nothing but grep and pattern matching.

No query planner. No schema lock-in. Just facts, stored predictably.

LISP Wiki 1.0 — The symbolic foundation

LISP Wiki ingests from anywhere: web clips, YouTube transcripts, arXiv papers, audio transcripts, spreadsheets. All of it gets compressed into one canonical representation: S-expressions.

The beauty here: I don’t need to run LISP as a programming language. S-expressions are purely a data format. The architecture is lightweight and local-first. An LLM (local or remote) browses the wiki’s Markdown views through Obsidian. Ingested materials live immutably in a raw/ folder. Retrieval runs on qmd—a hybrid BM25 + vector search—with embeddings powered by a local model, so no query ever leaves my machine.

The six node types

At the core is knowledge.lisp: a flat file of S-expressions that is the single source of truth. Markdown pages are derived views. The graph is the canon.

Six node types encode everything the wiki knows:

• Source: provenance — where a piece of knowledge came from
• Entity: a real-world referent (person, organization, product, system)
• Concept: an idea or abstraction
• Claim: a single declarative assertion with a confidence level
• Event: a timestamped occurrence
• Rel: a typed, weighted edge between any two nodes

Here’s what it looks like:

;; A concept node — relations stored as data, not in a join table
(concept :id "reverse-curse"
         :label "Reverse Curse"
         :summary "LLM failure to infer B→A after training only on A→B."
         :relations ((:type mitigated-by :to "in-context-learning" :weight 0.8)
                     (:type mitigated-by :to "neuro-symbolic-ai" :weight 0.6))
         :sources ("closer-look-transformers-ts-2025")
         :tags (llm-limitation))

;; An event node — date is structured data, not buried in prose
(event :id "dreamforce-2023-einstein-launch"
       :label "Dreamforce 2023 — Einstein 1 Platform Launch"
       :date "2023-09"
       :summary "Marc Benioff and Parker Harris unveiled the Einstein Trust Layer."
       :sources ("salesforce-einstein-trust-layer-forbes")
       :tags (ai-trust enterprise))

This structure does something flat Markdown can’t: it makes relationships queryable. A search isn’t a text match; it’s a graph traversal.

Adding procedural intelligence

Two additional node types live alongside the data: rules and domain lambdas (defun). These are not tools — they’re facts about how your domain works.

A rule is an executable lint check: a pattern, a condition, and an action that fires when the pattern matches.

(rule :id "rule-contradicting-high-confidence"
      :label "High-confidence claims in mutual contradiction"
      :severity :important
      :pattern "Two :high claims where one :supports and the other :contradicts the same node."
      :action "Flag each pair. Never auto-resolve — surface for human review.")

Rules are not documentation — they’re checked mechanically on every lint run. My wiki audits itself.

A defun encodes a calculation intrinsic to a specific domain. For example, a research ROI multiplier or a budget reallocation formula. Expressed as a lambda directly in the graph.

This is where LISP Wiki diverges sharply from conventional knowledge bases.

Operational queries on real data

Raw data files — campaign spreadsheets, benchmark CSVs, multi-week metric exports — live immutably in raw/. They're never loaded into the graph as rows or tables. The graph doesn't store data; it stores what the data means.

When a query requires quantitative analysis (“Which channel had the best ROAS in week 2?” or “Reallocate budget to the top performer”), the agent follows a procedural workflow:

1. Search the graph for relevant defun nodes—analytical recipes encoded as lambdas.
2. Execute the recipe against the raw file (a temporary, read-only Python pass against the CSV).
3. File only the high-signal output back into the graph: (claim ...) nodes with confidence levels, (event ...) nodes for summaries, and new relations.

The raw numbers never enter knowledge.lisp. What enters is the decision: a claim like "ROAS for paid-search exceeded display by 2.3× in week 2 (:confidence :high)" with full source provenance. Future queries retrieve that claim in milliseconds via grep, without re-running the analysis.

This is write-time synthesis. Heavy computation happens once at ingest. All subsequent queries are cheap reads against compact symbolic nodes.

The query toolbox

Graph traversal logic lives separately, in skills/query-helpers.lisp. Knowledge and tools are never mixed.

Because every node has a predictable S-expression shape, a simple grep "^(rel " followed by a field filter is a complete, correct query. No query planner needed. The helpers formalize this into reusable operations:

;; All nodes reachable from "timegpt" in 2 hops,
;; following only 'implements and 'used-by edges,
;; restricted to nodes tagged 'forecasting
(multi-hop "timegpt" 2
           :relation-filter '(implements used-by)
           :tag-filter '(forecasting))

;; Every contradiction in the graph — both sides with source provenance
(find-contradictions)

;; The 5 strongest outgoing connections from a node
(top-relations-by-weight 5 :from-id "itransformer")

None of these require a Lisp runtime. Each is agent-simulated as a sequence of grep passes, assembled into structured output. The uniform node shape makes it reliable.

Key principle: knowledge.lisp holds what is true. query-helpers.lisp holds how to ask questions about it. Data never leaks into tools; tool logic never contaminates the canonical store.

LISP Wiki 2.0 — Compounding intelligence

The first evaluation run — 10 gold-standard questions — confirmed the query model worked. Instead of linear document search, the system touched only the minimal relevant working set. The architecture held.

But that raised a harder question: if progressive context loading works for single queries, what happens when I run a full agent swarm inside the vault?

The hypothesis: LISP Wiki’s subtree-loading design could act as a context substrate for multi-agent work. Each agent loads only the subgraph relevant to its role, freeing up the bulk of the context window for thinking and reasoning rather than retrieval.

The architecture enforces one hard invariant: no child agent may write to knowledge.lisp directly. All updates flow through a single filing workflow. The knowledge graph stays consistent. Agents stay coordinated without requiring locks or a central arbiter.

Two test runs

Run 1: A swarm against 14-day marketing channel data. Run 2: The same swarm against a time-series forecasting paper.

In both cases, agents followed the protocol: analysis was grounded in data and papers that were in scope from the start. Mathematical work stayed within bounds.

The rough edges were real. Agents made unnecessary tool calls — a design problem, not a knowledge problem. Sequential tasks pulled in several hundred lines of Python source that couldn’t be pre-filtered. Neither was fatal, but both were costs.

What was promising: the swarm delivered a novel concept supported by data. It got there by stepping outside the conventional analytical frame.

The compounding insight

An agent swarm is not a cost-saving measure in rounds one or two. The overhead is real. Design problems surface early. The runs are expensive.

The case for it is compounding. Each run deposits new claims, new relations, and new findings into the graph. The vault gets smarter. Later runs start from a richer base and cost less to ground.

That is what transforms a personal knowledge vault into a home lab: not automation for its own sake, but a workspace where routine analysis and edge research share the same canonical store. A place where the cost of answering hard questions decreases over time rather than resetting with each session.

Verdict

The whole project answered two specific questions I had at the start:

1. Does the symbolic knowledge layer work? Yes. Evals confirm a measurable edge in retrieval speed, context efficiency, and answer quality. (The numbers are below.)
2. Can Obsidian become an AI-queryable personal knowledge vault? Yes. My agent swarm runs inside the environment, turning it into a complete home lab. With a local LLM (e.g., Qwen2.5–32B-Instruct for reasoning + tool use), LISP as the canonical truth layer, and local embeddings, the entire system runs privately on my desktop. Obsidian is free. LISP syntax compounds with scale.

The proof

4.1× fewer tokens. Vanilla wiki averages 38,295 tokens per query. LISP averages 9,360.

+83% quality lift. Judge score: 2.2 vs 1.2. LISP wins 8 of 10 query types. The failed 2 are harness cap issues.

Query wins:

• Single-hop relation: 7.5× fewer tokens
• Entity-event lookup: 7.4× fewer tokens
• Tag-scoped set: 10.3× fewer tokens
• Multi-source synthesis: Vanilla returned empty. LISP scored 3/3.

The scores proved the LISP wiki graph shines.

Use cases

LISP Wiki works best when know-how and show-how live together. Here’s what LISP Wiki is capable of:

Use case 1: Weekly social media analytics

Save CSV/XLSX files from social channels: engagement, reach, impressions, spend. Agent ingests the data once. (defun social-roi ...) runs analysis. Findings become claims: "Instagram videos under 2min outperformed carousel by 1.8× engagement (:confidence :high)". Filed in the graph.

Next week: new data arrives. Agent searches for “Instagram video performance.” Finds last week’s claim in milliseconds. Compares. Reports the delta and trend. By week 4, patterns emerge that only show up after querying the same question three times. The knowledge compounds inside knowledge.lisp.

Use case 2: Research without amnesia

I track the AI space as a hobbyist: clip articles, save papers, dump transcripts into the vault. Each paper becomes a node. Relations encode: “Paper A cites Paper B,” “Concept X appears in both A and C.”

Query: (multi-hop "vector-embeddings" 2 :relation-filter '(cites implements)) finds all concepts reachable from embeddings in 2 hops. Thesis, antithesis, and synthesis all land in one place. I can see patterns and rabbit holes I'd miss without the graph.

Setup and maintenance cost: minimal. Everything runs locally.

What’s next

There’s more to build on LISP wiki:

• Full local environment. End-to-end ingest on-device, multi-media support, visual section for data analytics.
• Serendipitous discovery. The classic textbook topic: Serendipity vs Signal. Can the system surface unexpected patterns and connections without explicit queries? Can it find novel combinations (like the pairing of LISP and wiki) that emerge from the knowledge graph itself, rather than only what we explicitly ask for?
• Syntax-level deduplication. Moving from document-level matching to catching duplicate nodes that look different on the surface but resolve to the same S-expression structure.

A personal note

The common framing of AI productivity is that AI does the heavy lifting and humans finish the last mile — a 90/10 split. But it’s a reverspective trick that deceives the viewer. The truth is a flip: when one inverts their angle, one can see the real image.

AI lowers the floor — it can produce 90% of something faster than humans can. I fed Karpathy’s original gist plus a short instruction to add a symbolic LISP layer to an AI, and got 90% of LISP Wiki 1.0 back almost immediately.

But the remaining 10% is not a finishing touch. It is the beginning of the project — where it dragged me into the rabbit hole.

What looks like a ready-to-unpack knowledge system is actually an assembly of atomic parts. Each has its own complexities and needs fine-tuning. When I asked AI why the token count ramped up, it gave me anything except an embedding issue. I wasn’t even aware QMD search needed my API key, and without it, the system silently chose a costly path. Sometimes its logic is hard to accept. Not all of those decisions are technical — they need human calibration.

The whole process became a collection of “You are spot on,” “Exactly,” “Good catch,” “You’ve nailed it.” These affirmations are sine qua non in human-AI collaboration — the human calibration that makes the work real.

But there’s something beneath the frictionless surface: teaming with AI lacks the sweat, tears, and coffee-sharing moments that human collaboration. There’s no friction, but there’s also no lived texture. Yet that’s precisely why the human voice — the judgment calls, the hesitation, the “wait, this doesn’t feel right” — becomes irreplaceable.

LISP #AI #KnowledgeManagement #PersonalWiki #Obsidian