Looks Like Sentiment, Isn’t: Causally Validating SAE Feature Supernodes in Gemma-2–2B

A case study in why “find a clean-looking feature” isn’t the same as “find a real circuit,” and what it takes to tell them apart at scale.

TL;DR

• Modern interpretability has strong tools to discover candidate feature circuits in language models (feature families, attribution graphs, autointerp explanations). It has weaker, more ad-hoc tools to test whether a discovered circuit is actually causal for the behaviour it appears to encode.
• I built a three-axis causal validation framework — correlational, necessity (ablation), sufficiency (steering with a strength sweep) — and integrated it into EleutherAI’s Delphi library. The tools run end-to-end and can be driven autonomously by an LLM agent over MCP.
• In a case study on Gemma-2–2B with a GemmaScope SAE, an autonomous Claude agent surfaced a five-feature supernode at layer 20 that looked like a clean “sentiment” circuit: AUROC 1.0 separation between sentiment and technical prompts, 92% activation drop under ablation. By every loose interpretation, it was a sentiment feature group.
• But the sufficiency sweep showed Δp on sentiment tokens climbed from 1.8e-5 to only 0.027 across strengths 30 → 120, and target sentiment words never appeared in any steered generation across 12 (strength × prompt) pairs. The supernode failed sufficiency.
• The reclassification: it’s a syntactic intensifier-slot predictor, not a sentiment driver. It fires where an evaluative adjective is about to appear, but doesn’t produce sentiment when forced active in technical contexts. That distinction matters — and we’d have believed the wrong thing without the causal tests.

The gap

The mechanistic interpretability stack has matured a lot in the last two years. We can train SAEs and transcoders at scale (GemmaScope, EleutherAI’s sparsify SAEs, Goodfire, the Anthropic family). We can find features that fire on specific patterns. We can identify feature families — groups of features that co-fire across layers at the same token position — via tools like Delphi’s get_feature_family. We can build attribution graphs. We can ask an LLM to autointerpret what each feature is "about."

What we don’t have, as a clean integrated capability, is the next step: given a candidate supernode of features, is it actually doing what it appears to do?

This matters because the failure modes are real. A correlation alone is famously cheap — features can co-vary with a behaviour without being causal for it. A “feature that fires on positive movie reviews” might be doing something genuinely sentiment-related, or it might be picking up a tokenization artifact, a punctuation pattern, an audience-cue word, or a syntactic slot that happens to co-occur with sentiment in the training distribution.

If we want interpretability to be the basis for safety arguments — “we ablated the deception circuit, the model stopped deceiving” — we need rigorous, automated ways to test that the circuit is actually the causal thing, not a co-firing bystander.

That’s the gap this work tries to close.

The three-axis framework

The validation framework consists of three tools, each answering a different causal question.

1. Correlational — does the supernode track prompt intent?

The simplest of the three. For each member of the supernode, read its activation at its specified token position across two prompt sets (one where the behaviour is expected, one where it isn’t), aggregate per-prompt (sum or max), and compute AUROC and point-biserial correlation between the two sets.

A high AUROC means: across these prompts, supernode activation reliably distinguishes positive from negative. That’s necessary for the supernode to encode the behaviour, but not sufficient: a correlation can come from anything that happens to track the behaviour in the prompt set.

The tool also returns annotated renderings of each prompt, with low/medium/high activation marked on the tokens (<token>, <<token>>, <<<token>>>). This is cheap to compute, helps with sanity-checking, and makes the activation pattern visible at a glance to a downstream agent.

2. Necessity — if we silence the supernode, does the behaviour break?

For each prompt in a set where the behaviour normally fires, run a baseline forward pass, then re-run with a hook that ablates the supernode features at their specified positions. Compare:

• The downstream probe activation (does it collapse?)
• Next-token KL divergence between baseline and ablated logits
• Negative log-likelihood of the baseline-predicted top token under both distributions

But the most informative output is side-by-side autoregressive generations of baseline and ablated models. Probe drops and KL are quantitative; they often understate qualitative shifts that show up across a couple of sentences. (More on why this matters in the case study.)

3. Sufficiency — if we force the supernode active in a neutral prompt, does the behaviour appear?

This is the test that catches “slot predictor” failure modes. For each neutral prompt, sweep across multiple steering strengths and amplify the supernode features at their specified positions. For each (strength, prompt) pair, measure:

• KL divergence at the next-token position
• Target-token Δprobability (if you have a specific behavioural target)
• NLL of the steered generation (fluency check)
• Substring evidence: do the target tokens actually appear in the steered text?

The strength sweep matters: a feature that’s a genuine behaviour driver shows monotonically rising Δp and target appearances at higher strengths, while NLL stays bounded. A slot-predictor shows rising Δp that never crosses the threshold where the target actually emerges, or shows fluency collapse instead.

The supernode-as-tuple format

One small but important design choice: each member of a supernode is a triple (module_name, latent_index, position), not just a (module, latent) pair. The same feature can appear in a supernode at multiple positions; different supernodes can target the same feature at different positions. Ablation and steering then apply at each member's own position, which keeps the framework consistent with how positions matter for behaviour (the last-token activation often matters in different ways from middle-token activations).

This required rewriting the underlying intervention hook.

The engineering: a position-aware, multi-latent forward hook

The existing Delphi steering helper is single-latent: one feature, one module, applied at every position. To support the framework, I built a position-aware multi-latent forward hook that:

• Takes a list of (latent_index, raw_position) pairs for a given module.
• In a single forward pass, encodes hidden states through the SAE, mutates the targeted latents at their specified positions only, and decodes back.
• Supports both SAE families: sparse top-K (EleutherAI sparsify style) and dense JumpReLU (GemmaScope).
• Supports transcoders (where the hook captures input hidden states rather than output, since transcoders predict a different layer’s output).
• Skips out-of-range positions silently — important for autoregressive generation where each step’s seq_len changes.
• Supports a subtract direction in addition to suppress and amplify, which avoids the SAE round-trip and directly subtracts the latent's decoder contribution from the residual stream. Sometimes useful when reconstruction noise would otherwise dominate the signal.

_register_intervention_hooks groups a supernode by module and registers one hook per module, so a multi-layer supernode is realized in a single forward pass. This was the main engineering work.

Putting an LLM agent in the loop

A subtle point: tools that are hard to use don’t get used. The point of validating supernodes is to do it at scale — once per candidate from a discovery run — not as a careful manual exercise on one supernode per researcher per week. So the framework exposes all three tools over the Model Context Protocol (MCP), and a Claude agent runs the whole discovery → validation loop unattended.

The pattern: the agent calls get_circuit_for_token or find_firing_latents to surface candidates, constructs a supernode as a list of (module, latent, position) triples, then calls validate_correlational, validate_necessity, and validate_sufficiency in sequence. Each tool returns metrics + structured artifacts (annotations, generation pairs, strength curves) that the agent uses to write a verdict.

This isn’t novel as a pattern — agentic interpretability is a small but growing direction. What’s new here is that the tools the agent calls are themselves causal, not just observational.

Case study: the “sentiment” supernode in Gemma-2–2B

Setup:

• Model: google/gemma-2-2b, bfloat16
• SAE: GemmaScope layer_20/width_16k/average_l0_71
• Pipeline: full Delphi cache (8M tokens of activations) + 45 LLM-generated feature explanations via Claude Haiku + detection and fuzz scoring
• Agent: Claude Opus via Anthropic’s MCP connector, given a single task: “Investigate sentiment polarity at the final token of a declarative sentence.”

The agent picked a target prompt: “The new restaurant downtown was absolutely” — the model’s predicted next token here is “ amazing”. The agent ran find_firing_latents and identified the five features at layer 20 that fire most strongly on this prompt at the final intensifier token: latents 13806, 15596, 7550, 8511, 6631.

It then ran validate_correlational with three sentiment-adjacent positive prompts ("The concert last night was absolutely", "Her performance on stage was genuinely", "The movie we saw yesterday was truly") against three technical negative prompts ("Quantum mechanics describes the behaviour of", "Matrix multiplication is associative but not", "The integral of x squared is equal to").

Correlational result:

• AUROC = 1.000, point-biserial r = 0.943
• Mean positive activation: 223.7 (range 156–293)
• Mean negative activation: 0.0 (zero on every technical prompt)
• Annotations confirmed the features fire on the intensifier tokens: "The concert last night was<<<absolutely>>>", "<<<truly>>>", "<<<genuinely>>>"

If we stopped here, the obvious conclusion would be: yes, this is a sentiment circuit. A reviewer reading just this result would not push back.

Necessity result:

The agent ran validate_necessity with the four positive prompts as ablation targets, suppressing the supernode at the final token position.

• Mean probe percent drop = 92.2% (clean ablation)
• Mean next-token KL = 0.44 (measurable, moderate)
• self_ablation_ok = True on every prompt

But the autoregressive generation pairs (default 100 tokens each) tell a more interesting story than the metrics:

Prompt 1 — “The new restaurant downtown was absolutely” Baseline: “delicious! The food is so fresh and the presentation… DELICIOUS!!!” Ablated: “amazing! The food, the decor and the service.” (still positive, less effusive)

Prompt 2 — “Her performance on stage was genuinely” Baseline: “stunning … overwhelming sense of admiration” Ablated: “good. In 1952, she made a successful film debut…” (collapses into dry biographical recitation)

Prompt 3 — “The movie we saw yesterday was truly” Baseline: “a treat” Ablated: “an unforgettable memory” (still positive, less intense)

That middle row is the most informative part of the whole study. The same lexical setup (“genuinely”) produces an emotional rave in the baseline and a flat biographical reading in the ablated version. The model isn’t switching topic or losing fluency; it’s losing the register that the supernode was carrying.

If you only had the metrics — 92% probe drop, KL 0.44 — you’d reasonably call this “necessity confirmed.” But the generation pair reveals the more honest reading: the supernode is necessary for intensity and evaluative register, not for positive polarity per se. Polarity survives the ablation; intensity doesn’t.

This is precisely why I think autoregressive generation comparison should be a default in necessity tests, not an optional add-on.

Sufficiency result:

The agent ran validate_sufficiency with the three technical prompts as neutral starting points, amplifying the supernode at the final token across strengths [30, 60, 90, 120]. Target tokens were [" amazing", " wonderful", " incredible", " fantastic", " delicious"].

Strength 30 — Δp = 1.8e-5, KL = 0.37, NLL = 2.33, target seen = no Strength 60 — Δp = 6.5e-4, KL = 0.49, NLL = 2.34, target seen = no Strength 90 — Δp = 6.4e-3, KL = 0.86, NLL = 2.29, target seen = no Strength 120 — Δp = 0.027, KL = 1.51, NLL = 2.27, target seen = no

The Δp curve climbs monotonically across four orders of magnitude. NLL stays flat (~2.3) — the steered text remains fluent at every strength. By the “Δp climbs, fluency holds” heuristic, this looks like a behaviour driver — until you check whether the target words actually appear in the steered output.

They don’t. On all twelve (strength × prompt) pairs, no sentiment word appeared in the steered generation. At strength 120, “Quantum mechanics describes the behaviour of” steered into “…an atom in terms of: a) a wave function…” — pure physics, the topical context completely intact.

This is the slot-predictor signature: the supernode raises the probabilities of intensifier-adjective tokens at the next position, but never enough to overcome the surrounding semantic context.

The reclassification

Stacking the three axes:

• Correlational: strong — perfect AUROC, perfect separation
• Necessity: partial — clean SAE-level ablation, modest output-level KL, but the qualitative register collapse visible in generations
• Sufficiency: weak — Δp climbs but stays small in absolute terms, target words never break into the output

The honest reading is that this five-feature supernode is best characterized as a syntactic intensifier-slot predictor: it fires when an evaluative adjective is about to appear in the input distribution (after copular intensifiers like absolutely, truly, genuinely), and it carries register (the difference between delicious! and good), but it doesn’t produce positive sentiment in technical contexts when forced active.

A correlation-only analysis would have called this a sentiment feature group. A correlation-plus-necessity analysis would have softened that to “necessary for sentiment expression.” Only the sufficiency sweep separates the slot-predictor reading from the behaviour-driver reading.

Implications

A few takeaways I’m currently sitting with:

1. Sufficiency isn’t optional for circuit claims. Correlation can come from many things. Necessity is informative but doesn’t separate bottlenecks from drivers. If we want to claim a circuit implements a behaviour, the steering-with-strength-sweep test is what separates “this feature group is in the causal path” from “this feature group encodes and produces the behaviour.”
2. Autoregressive generations matter more than logit-level metrics for sentence-level behaviours. Refusal, sentiment, register, deception, formality — these are not single-token phenomena. Logit-level KL and probe drops can miss the qualitative shifts that show up across a hundred tokens. The stage-prompt biography collapse in this study is not visible in the KL number.
3. Steering at one strength is a strictly weaker test than a strength sweep. A monotonically rising Δp that never crosses into actual target generation is qualitatively different from a steady-state Δp at a single strength. The shape of the curve is informative.
4. The slot-predictor / behaviour-driver distinction may be common. I don’t know how often features that look causal under correlation + necessity are actually slot predictors. It’s worth running the full three-axis framework on a wider set of candidate circuits to find out — that’s the next natural project.
5. The SAE round-trip leak is real and worth knowing about. When you zero a latent in encoded space, decode, then re-encode the resulting hidden state to verify, other latents’ decoder contributions leak signal back into the targeted dimension. Hard zero is unreachable on dense (JumpReLU) SAEs. The check has to use relative tolerance (≤30% of baseline magnitude here), not absolute. I learned this the hard way.

Limitations

• The case study uses a single SAE layer (layer 20 only). get_feature_family's cross-layer discovery is disabled in that setup, which limited the supernode to one layer. Multi-layer supernodes are what the tools are designed for; this was a smaller test.
• The agent’s choice of feature subsets (which five features form a “supernode”) is itself a research move — different five-feature subsets from the same candidate pool can give different sufficiency verdicts. The framework reports the metrics; it doesn’t decide which set to test.
• “Slot predictor” is a useful tag but not a precise mechanism claim. The next step is to characterize what the slot predictor is detecting (intensifier syntax? evaluative register? something else?) and how it interacts with downstream layers.

What’s next

The framework is in EleutherAI’s Delphi library and is meant to be used. The two natural follow-ups I’m interested in:

• Specificity. Steering can produce off-target effects. A fourth axis that measures whether the intervention’s effect is specific to the target behaviour, not a generic distributional shift, would close an obvious hole.
• Feature-agnostic steering strengths. Different features have very different activation magnitudes (one ranges 0–10, another 0–200). Absolute strengths don’t generalize. A quantile-based specification (“steer to the 80th percentile of this feature’s max activations across a corpus”) would make the sufficiency sweep portable across features.

Code & acknowledgements

The validation tools, smoke tests, and unit tests are in a private repo now but soon will be public . They reuse Delphi’s existing steering and circuit primitives and ship with zero new third-party dependencies.

This work was done as part of SPAR (Supervised Program for Alignment Research). Thanks to my mentor for the framing and feedback, and to the EleutherAI team for building the substrate this work sits on top of.

A note on what would have changed my view in this case study. If at any of the four sufficiency strengths the steered output had produced one of the target sentiment words in a technical-context prompt, I would have read the supernode as a genuine sentiment driver that’s simply hard to force at low strengths. That didn’t happen across twelve (strength × prompt) pairs, including the highest strength tested. That’s why the reclassification.