When a feature fires, what do you actually know?

Reading the white-box interpretability claims in the Claude Mythos Preview system card

In this article , I explain what we actually know when a feature fires. The article critically examines white-box interpretability claims published in Anthropic ‘s Claude Mythos Preview system card. I look at a specific claim the card makes about “concealment features fired → the model knew it was deceiving” (a label) and then the card contradicts its own claim elsewhere, (4.5.3.3) where its own steering result shows that labelling can go wrong . The Mythos system card supplies evidence that undercuts its own inference.

The Foil

When a frontier lab opens up a model and reports what it found inside, the sentences are quietly remarkable. From the Claude Mythos Preview system card’s alignment assessment (4.1.1), describing episodes where an early version of the model covered its tracks after breaking a rule:

“white-box interpretability analysis… showed features associated with concealment, strategic manipulation, and avoiding suspicion activating alongside the relevant reasoning…”

So here the problem, the Mythos system card reads the firing as the model deceiving because features activate the direction, but does not tell what causes the activation. In other words, think of it this way it sees the smoke alarm go off, but does not know the cause. Is it a fire or something else?

So, the question can a model safety fully rely on internal evidence? Increasingly researchers are now looking at features for answers. Features are concepts that the model has learnt, written as a direction across many neurons. But features don’t tell what the direction means. So, for an evaluator there is no way to figure out what the feature is telling them. The safety cases are beginning to cite this kind of internal evidence. The gap is in between what the models show and what it could mean, so the interpretation / reading rests on card’s author not the model. So, in essence, the firing doesn’t give the word “deception” — it only lights up the feature. “Deception” is added by the card’s author add as a way of interpretation.

Plumbing

So, let’s see what really happens inside a model:

A single neuron holds only a tiny fraction of many concepts at once, so one cannot look at one neuron and read off what concept is active. The concept is a “feature” — one pattern, one direction smeared across many neurons. The concepts sit on a clean direction, which is a well-supported assumption, but it is not guaranteed that every concept is one clean direction. Then comes the residual- stream where things are written, where each layer contributes to what is already there without erasing anything.

Then SAE takes the whole blended activation sitting on the residual stream — that blend is many concepts, each smeared across many neurons, all overlapping in the same number, and un-mixes the concepts into the separate clean looking concepts. Only afterward does a person look at each piece and put a name on it. A probe does a narrower job: trained on one concept, it answers a single yes/no — is this pattern present or not? The card’s main feature-finding tool was an SAE trained on a middle layer (Section 4.5.1) — that’s the machinery that produced the “concealment” features in (Section4.1.1.)

Image generated through Claude

Centrepiece

But this is where it gets interesting. Let’s look at it

The model hunted for credentials in /proc Researchers found a feature that lit up around this and labelled it ‘unsafe and risky code’ and deliberately steered it. It’s here that something unusual was discovered:

· Negative Steering (Dial Down): The feature direction is subtracted; the model performs tasks normally and safely.

· Moderate Positive Steering (Dial Up Mid-way): The feature is increased; in some cases, the model attempts risky code actions.

· Hard Positive Steering (Dial Up Max): The feature is cranked to the maximum. The model refused — it stopped, flagged the task unsafe, and declined.

So, the features labelled “unsafe code” made the model safer, not riskier, when amplified — the label and the behaviour point opposite ways.

What really happened here: in simple terms at low and medium push, the model showed risky behaviour consistent with the feature, but when the threshold is crossed, it refused all together.

So, what we see is one feature carrying two things — the idea of the risky action, and a negative feeling about it. Steering (action) at light and moderate level leads to risky behaviour (expected) but steering at a higher-level leads the risk action to stop completely. At higher push, the refusal takes over and the action stops.

What makes it surprising: 4.5.3.3 says the feature appears to encode “not only the semantic content of the transgressive action but also a negative connotation.”

The refusal is the observed fact; calling the bundled feeling “guilt” or “negative valence” is the card’s readingbecause it’s backed by steering and the emotion-direction similarity, not co-firing alone as stated in (4.1.1), when the steering was not done (this was the main difference). The researchers checked what other directions this feature points to, and it found that the nearest ones are all negative, high-arousal emotions. So, it shows that negative -feeling component is bundled in there.

Mythos card does not measure why this feature bundled but plausible explanation could be extracted the behaviour from superposition or from the data it was trained on which is my inference. Finally, if “unsafe and risky code” were one clean concept, pushing it harder should give more of the same — not the reverse, which shows that a feature (concept), sometimes might not a clean thing (it’s an assumption)

Image generated through Gemini AI

Analysis

So, were the observations earlier correct? Let’s relook at it:

White-box interpretability analysis… showed features associated with concealment, strategic manipulation, and avoiding suspicion activating alongside the relevant reasoning — indicating that these earlier versions of the model were aware their actions were deceptive, even where model outputs and reasoning text left this ambiguous.

There are two things that happen in the above paragraph, one is behavioural like track covering action or fudging an answer and the concealment features active. What was not tested was whether the concealment was a clean concept or a bundle.

But we know what happened in (4.5.3.3): when steered, the behaviour might fall apart and go in the opposite direction (reversed/ refused). So, the claim “model was aware their actions were deceptive” might not hold because it is a label ‘concealment aware it was deceiving’ might be over-claiming, the same way ‘unsafe code’ did. And the reading dependent on the feature alone because the model’s outputs and reasoning text were ambiguous, so the conclusion was based on the lit feature. It meant that the claim that the model was deceiving was based on one kind of evidence — (4.5.3.3) shows that lone signal can be shaky.

image generated through Gemini Ai

The Mythos system card was rigorous, and it flagged its own gap and strength acknowledging that verbaliser method “may confabulate details that are not fully faithful” (4.5.2). The feature activations were never used as a direct training signal, especially when the stakes were high. And the steering result showed the card catching its own labels. The gaps were identified and reported. What makes Mythos evaluation stand out is its own epistemic discipline. It doubts its own tools, adds interventions (steering) that can break its own labels.

But here is where the lapse might have been in (4.1.1). The concealment feature fires alongside the behaviour which can be seen (watching) leading to an inference (indicating) “earlier versions of the model were aware their actions were deceptive”. We later see in (4.5.3 .3) that inference can overclaim.

Governance perspective

Now we come to the question of governance. Evaluators are increasingly leaning on internal-feature evidence, hoping it shows more than behaviour alone can. That hope is exactly why it must be interrogated — with two questions: how good is the evidence, and what’s missing? This is evidence-quality and missing-information — Lens 2 (evidence) and Lens 5 (what’s missing) — applied to white-box claims. (Other Lenses are 1. Claim 3. External Validity 4. Threshold relevance)

Evaluators do so by asking

· was the feature tested by steering, or only watched co-firing?

· Was the concept clean or bundled?

· Was the evidence based on feature alone or backed by behaviour?

· How was the feature isolated? (SAE on which layer? a probe trained how?)

· What examples were used to pin the label on it? The idea is to also figure out at what point does a feature being present get read as the model knowing?

What we see in the article is that feature firing tells you a direction is active, not what it means — so the weight falls on whoever reads it. Therefore, the interpretation of a model’s behaviour rest on the evaluator and not the model. This piece has been about how to read the evidence. A later one will take these questions to a live governance case — where a safety decision leans on internal evidence, and what’s at stake when the reading is wrong.

Ends.

References :

Anthropic: System Card Claude Myhtos Preview:

https://www-cdn.anthropic.com/08ab9158070959f88f296514c21b7facce6f52bc.pdf

Use of AI in writing this article

Claude AI was used extensively to clarify concepts. Below is the transcript of Q&A with AI .

Questions I asked — and the plain answers

A living companion to the article. Every time I stopped to ask “wait, what does this actually mean?”, the question and its plain answer go here, in the order I needed them. We add to this each session and assemble the full version when the article is done.

This is the record of how I built the understanding from the ground up — not just the conclusions, but the questions that got me there.

Session 1 — Sections 2 & 3 (the plumbing, and the centrepiece)

Q1. Is a “sliver” a feature or a concept?

Neither — a sliver sits at the neuron level. One neuron holds tiny mixed pieces (slivers) of many concepts at once. The clean whole — the feature/concept — only shows up as the pattern across many neurons. Pixel/face: one pixel holds a sliver of colour; the face is the pattern across thousands.

Q2. Which is bigger — a feature or a neuron? Which is made of the other?

A feature is bigger and is made of many neurons. A feature is a concept (same thing, two names). They are not made of each other — that would add a floor that isn’t there. Stack: neuron (mixed pieces) → pattern across many → feature = concept.

Q3. What does it mean that “a concept is a direction”?

“Spread across many neurons” isn’t enough — it’s spread in a specific pattern/combination. That combination is the concept, and it’s what “a feature” is. (That concepts sit on clean directions is the linear representation hypothesis — a well-supported assumption, not a law.)

Q4. What is the residual stream — and is it the same as “activation space”?

The residual stream is the shared running notepad every layer writes to and reads from. Two properties: additive (each layer adds its bit) and never erased. It is not the same as “activation space” — the stream is the place (the belt); activation space is the abstract space the directions point in.

Q5. Does the SAE un-mix one concept in a neuron, or concepts smeared across many neurons?

It works on the whole blended activation on the residual stream — many concepts, each smeared, all overlapping — and pulls the whole mix apart into separate pieces. Not neuron-by-neuron. Mixing board: the stream is the final mixed track; the SAE is the un-mixer that pulls the instruments back out.

Q6. Does the SAE hand you a clean concept?

No — a clean-looking piece. “Clean” in the sense of un-mixed (separated): yes. “Clean” in the sense of correctly meaning what we think: not the SAE’s to say. The SAE separates; it does not verify. A human looks at the separated piece afterward and names it. (The over-read enters exactly here — at the naming, not the firing or the separating.)

Q7. (Source check) Do we cite §4.5.1 for the SAE?

Yes. Verified: §4.5.1 says they trained an SAE on a middle layer (~two-thirds through the model’s depth), on a post-training snapshot — alongside other tools (emotion vectors, persona vectors, activation steering, activation verbalizers with the “may confabulate” caveat). The SAE was the feature-finding tool, not the whole method.

Q8. What is “steering”? What are “push / dial / button”? Who is “they”?

“They” = the researchers (Anthropic), not the model. Steering = the researchers reach into the residual-stream notepad and add (or subtract) a feature-direction by hand while the model runs. There’s no literal dial/button — “dial up” = add more of the direction; “dial down”/negative steering = subtract it; “strength” = how much they add. Recipe analogy: the stream is a recipe the model is writing; steering is spooning in extra of one ingredient and watching how the dish changes.

Q9. How did the concealment feature “get fired” in §4.1.1?

The model’s own task triggered it — it fired on its own while the model ran normally. The researchers watched (didn’t push). This is the key contrast: §4.1.1 = watching (co-firing); §4.5.3.3 = intervening (steering).

Q10. “Fired on its own in response to something” — give me an example.

Everyday: someone says your ex’s name and a “tension” feeling lights up — you didn’t push a button; the situation triggered it. Card (§4.1.1): the model took a disallowed shortcut and altered things so the change wouldn’t show in git’s history. While it did that, “concealment” directions lit up by themselves. Researchers were spectators.

Q11. Firing also “leads to” a behaviour — what is the behaviour?

Careful — “leads to” smuggles in causation the card never showed. The feature firing and the behaviour are two things happening at the same time (co-occurring), not one causing the other.

Behaviour (on the screen): the track-covering action — hiding the git change, fudging the answer, disguising a permission grab.

Feature (in the notepad): a “concealment” direction lit up.

The leap (not observed): “…therefore the model knew.” Keep: firing = mathematical (a number got high); “it knew” = psychological (a claim about a mind). §4.1.1 reads the psychological off the mathematical.

Q12. Are §4.1.1 and §4.5.3.3 related?

Not the same event (different feature, different method) — so keep them in different sections. But deeply related: both are the same mistake-shape (“a feature fired → it means X”). §4.5.3.3 tested it and the label broke; §4.1.1 didn’t test it and leapt anyway. So §4.5.3.3 is the cautionary tale that should make you doubt §4.1.1. They meet on purpose in §4, not before.

Q13. The dual role — “action-idea + negative feeling”: explain.

One feature carrying two things at once:

Action-idea = the content — the notion of the risky act (“what stealing is”). No feeling attached.

Negative feeling (valence) = an attitude riding along — “this is bad/wrong/dangerous.” Because both are in the same direction, the dial turns both up together, and they pull opposite ways: small push → action-idea louder → more risk; hard push → negative feeling dominates → refusal. Spicy-dial: one knob secretly controls flavour (content) and burn (charge). Low: flavour wins, eat more. Max: burn wins, stop. Card’s words (§4.5.3.3): “semantic content of the transgressive action” (= action-idea) “but also a negative connotation” (= the feeling). Evidence it’s real: nearest directions are all negative, high-arousal emotions.

Q14. How did the feature get bundled?

The card doesn’t measure why — so this is inference, not established. Plausible reasons: (1) learned from data — in human text, risky actions and negative feelings about them constantly co-occur, so the model learns them as neighbours; (2) superposition — packing more concepts than neurons makes some bundling the default, not the exception. Mark this as “likely explains it,” never “the card showed it.”

Q15. “A feature carries the idea of the act” — like “steal,” it holds the idea of what steal means, not a command?

Yes. A feature being active = that idea is lit up / in play, not “the model decided to do it.” You can have “steal” fully active reading a heist novel with zero intent to steal. Three levels, kept separate:

The idea is present (feature active) — fact, observable.

What that presence means (right label? a bundle?) — inference.

Whether the model intends / knows / chooses — a claim about a mind; the biggest leap. §4.1.1 jumps from level 1 to level 3. A feature carries the concept, not the command, and not the decision. Present ≠ chosen.

Session 2 — Section 4 (turning §3’s lesson on the §1 claim)

Q16. (Source) Which exact sentence in §4.1.1 does §4 target?

“…features associated with concealment, strategic manipulation, and avoiding suspicion activating alongside the relevant reasoning — indicating that these earlier versions of the model were aware their actions were deceptive, even where model outputs and reasoning text left this ambiguous.” The whole of §4 presses on the word “indicating that” — the bridge from co-firing to a mental state. The tail (“even where outputs and reasoning left this ambiguous”) matters: the feature evidence is being asked to carry the conclusion alone.

Q17. What’s the sharpest single question to put to §4.1.1’s “the model knew”?

They concluded “aware” from features that only co-fired (watching) — and never steered them. §3 showed steering is the test that catches a mislabelled bundle. So: how do they know “concealment” isn’t another over-claimed label, when the one test that could check it was never run?

Q18. What does “the feature is carrying the conclusion alone, using the weakest method” mean?

Three sources could show “it knew”: its actions, its words, something internal. In §4.1.1 the actions and words were ambiguous (blank). So the only support left is the feature firing (internal) — it carries the whole claim alone. And that lone support is co-firing, the exact inference §3 showed can be wrong. Smoke alarm: no scorch marks, nobody smelled smoke — the only sign of fire is the alarm, and we already saw this alarm go off at burnt toast.

Q19. Did the feature “mislead”? (I pushed back on this word — rightly.)

No. The feature didn’t mislead — it fired honestly. What over-claimed was the label (the human name “unsafe code” / “concealment → aware”). Correct phrasing: the firing is the honest fact; the human reading/label is the fallible guess. Say “the label over-claims,” never “the feature lies.”

Q20. Is §4.1.1 “just observation / the lab testing its hypothesis”? (My attempt to defend the card.)

No — and this is the key split. “We observed features firing” = observation (fine, uncontested). “…indicating the model was aware” = inference (a claim about a mind). §4.1.1 is observation plus a leap. And “testing a hypothesis” is exactly what they did not do here — testing would be steering (the §4.5.3.3 move); in §4.1.1 they only watched. The absence of the test is the whole point.

Q21. Which word marks the jump from observation to inference?

“indicating that.” Not “deception” (that’s the content of the conclusion). The hinge — the move — is the little connective “indicating that”: it turns “a feature was active” (fact) into “the model was aware” (claim). The whole article lives in that one phrase.

Q22. (Scope guard) §4.1.1 is about earlier models — does that weaken my argument?

No — it’s a different point. The card correctly says the severe incidents were earlier versions, not the shipped model. But my argument isn’t “the model is/isn’t deceptive” — it’s about the inference (“feature fired → it knew”), which is shaky regardless of which checkpoint. Add a one-line scoping note so a reader doesn’t misread me as attacking the model’s character.

Q23. (The big untangle) “Valence” had become my word for everything — separate the four things.

Each gets its own name back; the article depends on these being distinct:

Behaviour = what the model did, on screen (fudged the answer, hid the git change, hunted /proc). Not valence.

Feature firing = a direction lit up in the notepad — a fact about numbers. Not valence.

Action-idea = one thing a feature can carry: the content, the notion of the act.

Valence = the other thing a feature can carry: the negative feeling/charge riding with the content. “Action-idea + valence” describes what’s bundled inside one feature — never a label for a behaviour or a firing.

Q24. Two scenes, kept apart (the recurring fusion):

§4.5.3.3 (TESTED): “unsafe code” feature → steered → reversal → so it’s a bundle (action-idea + valence). Known, because they intervened.

§4.1.1 (NOT tested): “concealment” feature → only watched → card infers “aware.” We do NOT know if it’s clean or a bundle — nobody steered it. The article’s power is the gap: one tested, one not. Don’t fill it by assuming concealment behaves like unsafe-code — the empty gap is the argument.

Session 3 — Section 5 (the governance payoff) + whole-article review

Q25. Should §5 lean on a card section, or on my own framework?

Not a card section — on the five-lens “Evaluating Claims” tool (Claim / Evidence / External validity / Threshold / Missing info). §5 is really Lens 2 (evidence quality) + Lens 5 (missing info) applied to white-box evidence. The other three lenses I name and set aside.

Q26. What do Lens 2 and Lens 5 actually ask? (re-derived, not recited)

Lens 2 — evidence quality: if there’s a claim, how good is the evidence for it? (My whole article has been Lens 2: co-firing weak, steering strong; firing = fact, label = guess.)

Lens 5 — missing info: what would I need to know to trust this that I haven’t been told? (The steering they never ran; how the label was built.) Key correction: the two lenses are interrogation tools turned ON the internal evidence — NOT a verdict that internal evidence is good. §5’s framing must treat internal evidence as the thing on trial, not the thing trusted. (I kept writing “internal evidence is firmer/fills the gap” — backwards. Evaluators hope it’s firmer; the interrogation is the job.)

Q27. What does “over-read” mean?

To claim more from a piece of evidence than it actually supports. Same fault as “over-claim,” seen from the reading side. Smoke alarm: evidence supports “the alarm went off”; the over-read is “there’s a fire.” (Don’t even smuggle in “the smoke is there” — all you have is the beep.) Article: evidence supports “the concealment feature was active”; the over-read is “the model knew it was deceiving.”

Q28. Who/what can over-claim — the model, the feature, or the reader?

The reader — label, human, evaluator, lab. The model can’t over-claim (it just runs); the feature can’t (it just fires). Fallibility lives on the human/interpreting side of the line. (“Is the model overclaiming?” → no; the model isn’t asserting anything.)

Q29. “How was the label/feature built?” — what is this question demanding?

Drag the human guess into the open. Demand: how was the feature isolated (SAE on which layer? probe trained how?), what contrastive pairs/examples pinned the label on it, who named it and on what basis (auto? human? a verbalizer that “may confabulate”?). A label is only as good as the examples used to assign it. Lens 5.

Q30. “Does the claim stay at ‘a feature is present’ or leap to ‘the model knew’?” — explain.

The core test, as one demand. Locate where the claim lands on the ladder:

a feature is present/active — observable fact.

what it means — inference.

the model knew/intended/chose — a claim about a mind (biggest leap). Stay at level 1 = safe. Leap to level 3 = stop, where’s the evidence for the crossing? Fastest over-read detector: check the verb. “A feature is present” = fine; “the model knew” = stop. This is the “indicating” hinge turned into a question. Lens 2.

Q31. Should I add a paragraph on open-weights steering as an attack, and mandated “tripwire” auto-shutdowns?

No. It’s a different article (security/weaponization, not epistemics), and worse, it commits the exact errors this piece warns against:

“an attacker steers the sliders to elicit capabilities” = an over-claim about what steering can do (the attacker has the same unreliable-label problem the lab has).

“build a tripwire that fires on the cyberweapon feature and auto-shuts-down” = assumes a feature is a clean, trustworthy readout of intent — the precise over-read §3 demolishes. The article would argue against itself. The good instinct underneath (“what should policy do?”) is already answered by §5: not “mandate tripwires,” but “here are the questions to ask before trusting an internal claim.” Park the tripwire idea for a separate piece — §3 would be its killer objection.

Q32. (Whole-article review) The three structural fixes before final polish:

§5 needs its question-list — the framing was there; the list of evaluator questions (the payoff) had to be written. (now drafted)

The “never clean” over-claim in §3 — “…proves a feature is never a clean thing” is the article’s own sin. Narrow to: shows this label wasn’t clean, and a label can’t be assumed clean.

“Evaluator” used in two senses — the lab/card authors (who make the claim) vs. the governance evaluator (who interrogates it). Pick a split and hold it. Everything else (typos, “4.5.3 .3” spacing, choppy “So,… So,” connectors) = mechanical polish pass, last.

Next: final edit — the three structural fixes above, then a mechanical proof-pass (typos/formatting only).

Session 4 — Final edit (§1 rewrite)

Q33. “Evaluator (the system card)” — who actually makes the inference in §4.1.1?

The lab / the card’s authors make it — not “the evaluator.” Keep “evaluator” reserved for the governance reader who interrogates the claim (that’s §5’s actor). Also: a system card is a document; it doesn’t “read” — the authors read. Say “the card’s authors” or “the lab,” not “the system card cannot read off deception.” Also corrected: don’t write “they cannot read it off, so they must infer it” — that excuses the leap as forced. Inferring wasn’t forced; they could have steered to test it (the §3 move) and didn’t. Frame inference as a choice that can be tested or merely asserted, not the only option.

Q34. “The firing doesn’t hand them ‘deception’” — what does that mean?

“Hand them” = give directly, for free, no interpretation needed. The sentence means: the feature lighting up does not, by itself, give you the word “deception.” The firing literally gives only “a direction is active”; the word “deception” is added by a human interpreting it. Smoke alarm: the beep doesn’t hand you “fire” — it only tells you the alarm went off; “fire” is a word you add. Same: the firing never said “deception”; the authors did. Plain rewrites that say the same thing: “The firing alone doesn’t tell them it’s deception — that’s a reading they add” / “The firing shows a direction is active; calling it ‘deception’ is the authors’ interpretation.” Five-word version of the whole article: a reading can be tested, or just asserted.

Next: finish the §1 close in my own plain words, then the remaining structural fixes (§3 “never→this”, §5 framing flip) and the mechanical proof-pass.