The Activation Test: How Reading an LLM’s Mind Reframed My Approach to Pricing Knowledge

Notes from a small experiment with Anthropic’s Natural Language Autoencoders.

The result I didn’t expect

Last week I ran a small experiment. I took 24 documents — six expert claims with concrete entities and numbers, six dense technical paragraphs, six pieces of marketing copy, and six pieces of plausible-sounding technical waffle — and fed them through a new interpretability technique to score how much “real knowledge” each one carried.

The result told me, with high confidence, that my marketing copy was denser than an oncology research paper.

This is not about a broken metric, but about why that surprising result was the most useful thing the experiment could have told me, and how it reframed my approach to one of the harder problems in building knowledge-pricing infrastructure: how do you programmatically detect signal from noise when every contributor wants to look like signal?

The problem I was trying to solve

I am working on a knowledge marketplace where contributors are paid in a native token for high-signal content. The economic mechanism mints a token when knowledge enters the network, with quality-adjusted rewards that scale with rarity and utility. This is the part of the system that has to be objective, programmatic, and resistant to gaming — and it is the bottleneck. Without it, every downstream mechanism (routing, decay, provenance, search) has nothing to anchor on.

Surface metrics — word count, readability scores, embedding similarity to “good” reference text — are all gameable. Anyone who has spent ten minutes inside an SEO playbook knows how cheap it is to dress up empty copy as authoritative content. I needed a metric that operates below the surface.

That is where the LLM mind-reader came in.

Natural Language Autoencoders, briefly

In late 2025, Anthropic published Natural Language Autoencoders (NLA) and released open-weights checkpoints for Qwen-2.5–7B, Gemma-3–12B, Gemma-3–27B, and Llama-3.3–70B. The setup:

Take a frozen LLM. Pick a layer somewhere two-thirds of the way through it — for Qwen-2.5–7B, that is layer 20 of 28. At any token position, the hidden state at that layer is a dense vector that encodes the model’s internal representation of the text’s meaning up to that point.

An NLA is two fine-tuned models on top of that frozen base:

• The Activation Verbalizer (AV) takes one of those vectors, injects it into its prompt at a specific token position, and generates a natural-language description of what the vector represents.
• The Activation Reconstructor (AR) takes that description back and produces a predicted activation vector. The cosine similarity between the original and the reconstructed vector measures how much information the description preserved.

The API ends up being almost embarrassingly compact:

from nla_inference import NLAClient, NLACritic

av = NLAClient("./ckpts/nla-av", sglang_url="http://localhost:30000")
ar = NLACritic("./ckpts/nla-ar", device="cuda:1")

explanation = av.generate(activation_vector)       # vector -> text
mse, cos    = ar.score(explanation, activation_vector)  # how well it round-trips

Think of an NLA as a domain-independent caption system for the inside of an LLM. Anthropic’s primary use case is interpretability. The hypothesis I wanted to test was that it could be repurposed as a measurement primitive: if cos(reconstructed, original) is high, then a few words preserved the content of the activation, so the activation must encode something a short description can capture — i.e., real, condensable knowledge. Filler would have nothing to capture and would reconstruct poorly.

That hypothesis was wrong, and the way it was wrong told me how to actually build the thing.

The setup

Four categories, six items each, length-balanced to ~40–60 tokens (Qwen tokenization). The extraction loop:

LAYER = 20
SKIP_FIRST = 10        # docs note: early positions haven't seen enough context

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-7B-Instruct")
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-7B-Instruct", dtype=torch.bfloat16, device_map="cuda:0",
).eval()

ids = tok(text, return_tensors="pt").to("cuda:0")
out = model(**ids, output_hidden_states=True, use_cache=False)
activations = out.hidden_states[LAYER][0]   # [seq_len, d_model=3584]

For each item, I sampled 6 positions evenly spaced through [SKIP_FIRST, n_tokens − 1]. 144 activations total. The four categories:

• S — Specific expert claims: concrete entities, numbers, mechanisms. “In stage III non-small-cell lung cancer with EGFR L858R, third-generation TKIs like osimertinib achieve median PFS around 18.9 months versus 10.2 months for first-generation gefitinib…”
• D — Dense technical paragraphs: formal definitions, math, computer-science mechanisms. “Compressed sensing recovers s-sparse signals from O(s log(n/s)) random measurements when the measurement matrix satisfies the restricted-isometry property…”
• F — Marketing filler: vague aspiration, no specifics. “In today’s fast-paced digital economy, organizations must embrace innovative solutions that drive transformative outcomes…”
• W — Plausible-sounding empty waffle: technical-flavored but content-free. “When designing distributed systems, it is important to consider scalability, reliability, and performance…”

For each activation, I ran the round trip — AV → text → AR → reconstructed vector → cosine against the original.

The first finding: density is the wrong primitive

  Category                 Mean reconstruction cos   Above noise floor  
 ------------------------ ------------------------- ------------------- 
  Marketing filler         0.914                     0.551              
  Empty waffle             0.917                     0.554              
  Specific expert claims   0.869                     0.506              
  Dense technical          0.844                     0.481

The noise floor is from scoring a deliberately unrelated explanation (“the quick brown fox jumps over the lazy dog”) against each activation: cos ≈ 0.36. Anything above that means the AV captured a genuine signal.

The two filler categories scored higher on reconstruction fidelity than the two knowledge categories, by a margin well outside per-asset noise.

Why? Because the AV was trained on FineWeb, which is dominated by marketing, business, and corporate-tech prose. Cliché-rich text is in-distribution for the AV — its repertoire of stock descriptors (“formal marketing tone”, “corporate communication structure”, “branding context”) slots cleanly onto activations that encode those patterns. Niche technical content — quanto options, post-tensioned slab specifications, agricultural-export logistics — is mildly out-of-distribution, and the AV’s descriptions for it are less informative for the AR to reconstruct from.

If I had shipped reconstruction cosine as the knowledge-pricing primitive, the system would have systematically rewarded filler over expertise. The opposite of what a knowledge marketplace needs. The kind of finding that, six months in, becomes a contributor exodus. Catching it now, on 24 documents, before committing to a mechanism, is the entire reason you run small experiments first.

The deeper finding: the fingerprint is in the words, not the geometry

Here is what saved the experiment. The AV does not only produce a similarity score — it produces a natural-language explanation. When I pulled out the actual content words the AV emitted for each asset’s activations, an obvious pattern appeared:

  Asset                                     Top content words AV produced from the activation       
 ----------------------------------------- -------------------------------------------------------- 
  Lung cancer + EGFR + osimertinib          medical, egfr, clinical, trial                          
  TLS 1.3 handshake                         handshake×13, client×11                                 
  Quanto options pricing                    currency×17, underlying×12, forward, rate, correlation  
  Post-tensioned seismic slabs              concrete×14, structural, engineering, design            
  Bayesian hierarchical models              variance×16, shrinkage×11, tradeoff, estimates          
  Compressed sensing                        sparse×11, recovery, sensing, matrix                    
  Diffusion models                          diffusion×13, models, square**                          
  Marketing copy (digital transformation)   innovation, transformation, disruptive                  
  Marketing copy (visionary disruption)     bold, disruptive, innovation                            
  Empty waffle (data governance)            data×36, privacy, compliance (surface echoes)

Every specific-knowledge item produced the right domain vocabulary. Not “medical content” — egfr. Not "finance" — quanto, correlation. Not "math" — shrinkage, sparse, recovery. The AV pulled out the precise terms you would use to ask a domain expert to summarise the topic.

Marketing copy produced marketing vocabulary. Empty waffle produced echoes of its own surface words — data×36 is the AV emitting the noun the input text repeated thirty-six times, with no enrichment.

The extraction is trivial:

import re
TOKEN_RE = re.compile(r"[A-Za-z][A-Za-z\-']{2,}")
GENERIC = {"format", "structure", "tone", "article", "blog", "marketing",
           "establishing", "implying", "describing", "introducing", ...}

def fingerprint(explanations: list[str]) -> list[str]:
    """Extract domain-content tokens from a set of AV explanations."""
    return [t.lower() for e in explanations
            for t in TOKEN_RE.findall(e)
            if len(t) >= 4 and t.lower() not in GENERIC]

This is the actual primitive. Reconstruction cosine is a misleading aggregate. The content of the explanations is a model-derived topic fingerprint, and it separates expertise from filler at a level the cosine cannot.

Why this is harder to game

A knowledge-pricing protocol needs adversarial robustness above all. A keyword-stuffing attack against an embedding-similarity scorer is trivial. A length attack against word count is trivial. A buzzword attack against any text-surface scorer is the entire business model of bad SEO.

Activation fingerprints have a property that surface metrics don’t: a contributor has to make the LLM internally represent the content for it to show up in the fingerprint. Pasting the word “quanto” into a marketing paragraph does not make the model’s layer-20 hidden state encode the geometry of quanto option pricing. The activation reflects what the text is actually about in the model’s representational space, not what surface tokens it contains.

This is not a perfect defense — adversarial inputs that genuinely manipulate model internals have been documented in the literature, and prompt-injection-style attacks against retrieval pipelines are an active area of research. But fingerprint-based scoring operates at a fundamentally deeper level than surface-text metrics, and the cost of mounting an attack scales with how good your model-of-the-model is. For most contributors, that is a wall. I will verify this empirically in the next experiment (see below).

What the mechanism wants to look like

The knowledge-pricing primitive I will build, then, is not a fidelity score. It is a fingerprint extractor with a rarity-weighted accumulator on top:

def asset_score(asset, base_model, nla, corpus_idf):
    """Pricing primitive: rarity-weighted sum of activation-derived content tokens."""
    activations = sample_activations(asset, base_model, layer=20, n=N)
    explanations = [nla.av.generate(a) for a in activations]
    tokens = fingerprint(explanations)
    return sum(corpus_idf.get(t, MAX_IDF) for t in tokens)

Where corpus_idf is an inverse-document-frequency table built over the marketplace's existing fingerprints (not raw text). A few properties follow naturally:

• Rarity-weighted by construction. osimertinib carries weight; transformation carries near-zero weight. The mechanism's "rarity × utility" curve is just IDF over fingerprints.
• Decay built in. As more contributors publish content with overlapping fingerprints, per-token IDF drops. “Stale knowledge yields less over time” arrives for free, computed against the actual marketplace state rather than wall-clock time.
• Provenance is a fingerprint overlap. Two assets whose fingerprints overlap heavily are almost certainly derivative of the same source, even if their surface text shares no n-grams. Attribution becomes computable.
• Search is the same primitive. A user query gets fingerprinted the same way; matching is fingerprint-against-fingerprint in the shared content-token space. Marketing copy does not rank for osimertinib because the activation cannot fake it.

The pricing and retrieval functions become a single mechanism. That is more architectural luck than I expected from a one-time experiment.

What this experiment didn’t tell

24 assets is a sample, not a study. Two specific caveats:

• Length confound. The filler items came out shorter than the expert items (~37 tokens vs ~55), which means part of the cos gap reflects positional depth rather than content.
• Hand-curated stopword list. The fingerprint analysis uses a small fixed GENERIC set ("format", "structure", "tone", …).

I tested one base model (Qwen-2.5–7B) at one layer (20). The four open-weight NLA checkpoints span 7B to 70B parameters, and fingerprint quality almost certainly scales with the base model size.

But the qualitative finding is robust enough to act on. The pricing primitive I will build is a fingerprint-rarity score with a continuously evolving corpus-statistics layer that natively handles the decay function.

What’s next

Things I will be running over the coming weeks:

1. Adversarial gaming test. Keyword-stuff a filler item with domain terms (egfr, quanto, compressed sensing) and verify the fingerprint stays in the filler cluster. If the fingerprint flips, the metric is fragile. This is the critical robustness experiment.
2. Length-matched controls. Re-run with strictly length-matched items per category, eliminating the positional-depth confound called out above.
3. Fingerprint stability under sampling. Same content, same activation positions, different sampling temperatures (T = 0.1, 0.5, 1.0). How much does the fingerprint move? Is there a temperature regime in which it is deterministic enough to serve as a content-addressable hash?
4. Multi-layer fingerprints. Layer 20 captures roughly content + format. Layer 5 captures syntax; layer 26 captures next-token predictions. Different layers should yield different fingerprints, and the union may be a more robust descriptor than any single layer.
5. Cross-model agreement. Fingerprint the same asset with NLAs over Qwen-7B, Gemma-12B, and Llama-70B. Where fingerprints agree, the signal is high; where they disagree, the asset encodes model-specific quirks rather than universal knowledge. This is a defensible “wisdom of crowds” denoiser.
6. Corpus-IDF construction. Build the actual IDF table from a 100k-document FineWeb sample, using the same NLA, so the scoring function lives against realistic marketplace statistics rather than a hand-curated stopword list.
7. Retrieval evaluation. Convert the fingerprint into a sparse retrieval primitive (BM25-style over content tokens) and measure recall against a labeled query set. If activation-derived retrieval beats raw-text retrieval on niche queries, this becomes a search story, not just a pricing story.

The deeper lesson is one I keep relearning: in any sufficiently new domain, the obvious metric is wrong, and the right metric is something the obvious metric poorly approximates. The wisdom economy will be full of these. Run the small experiments. Trust the surprising results.

The cos numbers were incorrect. The content tokens were the right ones.

Code & data. Everything in this post is reproducible on a 2 × 24 GB GPU server in under 30 minutes. Repository (corpus, extraction, round-trip, fingerprint analysis). Pull requests with stronger corpora, adversarial test cases, or experiments with larger models are very welcome.