Using the NPU to Do Something Useful (For Once)

I built a second brain for my Obsidian vault that runs entirely on the chip in my laptop that’s never done a day of work in its life.

Open Task Manager on a laptop bought in the last couple of years and you’ll probably see it: a little performance graph labeled NPU, sitting next to CPU and GPU, permanently flat at 0–1%.

That’s not a bug. It’s dedicated AI-inference silicon — Intel calls theirs the AI Boost NPU, AMD calls theirs Ryzen AI, Apple’s had a Neural Engine since 2017, Qualcomm ships one in every Snapdragon X chip — and almost none of it ever gets used, because almost no everyday software actually targets it. It sits there, fully paid for, doing nothing, while every “AI-powered” app you use routes your data to a GPU cluster in someone else’s data center instead.

I got annoyed enough about this to actually do something with mine. This is the story of what I built, and why I think “runs on the NPU” should be a much more common sentence than it currently is.

The actual problem I had

I keep an Obsidian vault. Hundreds of notes, PDFs, research papers, lecture transcripts, half-finished thoughts. It’s genuinely useful — right up until I need to find something in it.

I know I’ve read about a topic before. Finding it means grep-searching across dozens of files and mentally stitching fragments back into an argument I half-remember having with myself six months ago. Knowledge was scattered, and the vault was growing faster than my memory of what was in it.

So I built Second Brain — a system that sits on top of the vault and actually reads all of it, so I can just ask.

What it does

Three layers of intelligence, stacked:

1. Hybrid search. Every note and PDF gets chunked with markdown-section awareness (not naive character splitting — it respects your headings), embedded into vectors with FAISS, and also indexed for keyword search with BM25. Fused retrieval catches both “similar meaning” and “exact term” matches.

2. A compiled wiki. This is the part I’m most pleased with. Instead of the agent digging through raw, messy chunks every time, an LLM reads sources and distills them into structured, cross-referenced wiki articles — inspired by Andrej Karpathy’s approach to personal knowledge compilation. New information about a topic updates the existing article instead of creating a duplicate, so the wiki actually accumulates and refines over time instead of just growing.

3. A knowledge graph. Entities — people, concepts, tools, papers, projects — get extracted from every note and woven into a graph with typed relationships (implements, extends, contradicts, references…). Community detection clusters related entities into topic groups with auto-generated summaries, so a broad question like “what do I know about transformer architectures?” gets a synthesized answer instead of a pile of file names.

An agentic loop ties it together: the agent gets a question, picks from 14 tools (search the wiki, search raw chunks, walk the graph, follow links), gathers context across a few hops, and answers with inline [Source: ...] citations for every claim.

A Chrome extension feeds it — one click, or a right-click on any page, saves it and pipes it into the ingestion pipeline automatically.

Why the NPU, specifically

Here’s the part that made this more than “yet another RAG demo.”

Most “chat with your notes” tools assume the model lives in someone else’s data center. That’s a fine default for a SaaS product. It’s a bad default for a tool whose entire job is to read everything you’ve ever written — including the drafts, the half-formed opinions, the stuff you’d never paste into a chat window pointed at a third party.

And there’s a hardware angle that I think gets slept on: the compute to solve this privately is already sitting in your machine. NPUs are purpose-built for exactly this kind of inference — tensor and matrix-multiply throughput tuned for running neural nets efficiently, at low power, without spinning up your fans or your GPU. A 3–8B parameter model is more than capable of routing tool calls and synthesizing an answer from retrieved context — which is the actual hard part of a system like this. The retrieval pipeline (hybrid search, the compiled wiki, the graph) is doing the heavy lifting; the model just needs to reason over what’s already been found.

So the entire pipeline — agent loop, entity extraction, wiki compilation — runs locally through LEMONADE, an OpenAI-compatible inference server that talks straight to the NPU. What that buys me:

• Privacy that’s structural, not promised. Nothing leaves the machine on the default path. There’s no ToS to read, no “we don’t train on your data (probably).”
• No bill. No token metering, no rate limit, no surprise charge because I asked it to summarize the whole vault.
• Works offline. On a plane, wherever — the model’s already on disk.
• It’s fast enough, because the model’s job here is narrow and well-scoped.

But I didn’t want to make “local-only” a hard constraint, because sometimes you genuinely want a frontier model’s reasoning on a harder synthesis question, or your NPU’s busy doing something else. So the LLM backend is a swappable component, not a hardcoded assumption:

# llm_provider.py — the only place that knows how to build an LLM client
def get_provider_config(provider: str | None = None) -> ProviderConfig:
    resolved = resolve_provider(provider)  # "lemonade" or "gemini", with fallback
    if resolved == "gemini":
        return ProviderConfig(
            key="gemini",
            base_url=GEMINI_BASE_URL,   # Gemini's OpenAI-compatible endpoint
            api_key=GEMINI_API_KEY,
            model=GEMINI_MODEL,
            ingestion_model=GEMINI_INGESTION_MODEL,
        )
    return ProviderConfig(
        key="lemonade",
        base_url=LEMONADE_BASE_URL,  # http://127.0.0.1:.../v1 — the NPU
        api_key=LEMONADE_API_KEY,
        model=LEMONADE_MODEL,
        ingestion_model=LEMONADE_INGESTION_MODEL,
    )

Because Gemini exposes an OpenAI-compatible API, switching providers is just a different base_url and api_key — the entire tool-calling loop, streaming logic, and citation format is shared, unchanged, between the NPU and the cloud. One toggle in the UI, and the exact same agent is reasoning on a different chip.

Local-first doesn’t have to mean local-only. It should just mean local is the default, not the exception you have to go out of your way to configure.

What it actually looks like

Screenshots below are from my own running instance, over my own vault — real compiled output, not staged demo data.

Ask anything. It searches the compiled wiki first, then raw notes, then the knowledge graph.

Every claim traces back to a source note. No hallucinated confidence.

The wiki grows and refines itself as I add more notes — this is what the LLM distilled from raw sources, not the raw sources themselves.

Note to self before publishing: Medium won’t pull these in via copy-paste reliably — re-upload the three PNGs directly into the Medium editor at these three spots (they’re in rag/docs/screenshots/ in the repo).

Try it yourself

The whole thing is open source: github.com/Readyaddy/NPU_based_second_brain

It’s built for an Obsidian vault specifically, but the pieces — hybrid search, the wiki compiler, the knowledge graph, the provider-swappable agent — generalize to any pile of markdown and PDFs you want to make queryable. If you’ve got an NPU that’s never done a real day of work, and a stack of notes you keep meaning to revisit, it might be worth an evening.

If you build something on top of this, or find a laptop with an even lazier NPU than mine, I’d like to hear about it.