Can We Read a Small Language Model’s Thoughts Before It Answers?

I recently tried a small mechanistic interpretability experiment on Qwen/Qwen3–1.7B. The question was simple: before a language model gives its final answer, does it already contain some human-readable internal concepts?

For example, if I ask the model about a spider, does the concept “spider” appear inside the model before it says anything? If it needs to answer how many legs a spider has, does “eight” or “8” become visible somewhere inside the network before the final token is produced? This is the kind of question that feels almost philosophical, but here we can make it technical and testable.

This experiment was inspired by Anthropic’s Transformer Circuits paper, “Verbalizable Representations Form a Global Workspace in Language Models”. Their paper studies Claude-scale models and asks whether language models form internal representations that are not only useful for computation, but also readable as ordinary words. They use a method called the Jacobian lens, or J-lens, to decode internal model states and see which words are internally active before the model finally speaks.

The important thing is that Anthropic is not saying that language models are conscious. That would be a much bigger and much more dangerous claim. Their narrower point is technical: some internal representations in language models seem to behave like a shared workspace. They are available across different parts of the computation, they can affect future outputs, and they can sometimes be decoded into normal human words.

This connects loosely to Global Workspace Theory from cognitive science. In that theory, many specialised processes may run in parallel inside the mind, but some information gets written into a shared “workspace”. Once information is in that workspace, it becomes available for speech, planning, memory, and action. Anthropic uses this as an analogy. The model may have many hidden computations going on, but some concepts become globally useful and verbalizable.

I wanted to see whether a much smaller open model, Qwen 1.7B, shows a weak version of the same thing. This is not a claim that Qwen 1.7B has the same global workspace structure as Claude. It is a smaller experiment with a smaller model and a much simpler setup. But it is still useful because it tells us whether this kind of readable internal concept appears even in open small models.

The Simple Intuition

Normally, when we use an LLM, we only see the final answer. We ask, “Is this code safe or unsafe?” and the model says “unsafe”. But that final answer hides all the internal computation. The model may have noticed the dangerous part much earlier. It may have internally represented “os.system”, “user input”, “command”, and “unsafe” before producing the final word.

Think of it like a student solving a problem silently. We only hear the final answer, but if we could see the student’s rough notebook, we might find the intermediate steps. Maybe the student has already written “spider”, then “8 legs”, before saying the answer aloud. A J-lens style probe is like trying to read the model’s rough notebook.

Of course, a language model does not literally write English words inside itself. It stores information as high-dimensional vectors. But some of those vectors may point in directions that correspond to words the model could later say. If a hidden state points strongly in the “spider” direction, we may say that the spider concept is readable there. If it points toward “unsafe”, we may say that unsafe is internally active.

That is the main intuition. We are not just asking what the model finally outputs. We are asking what becomes readable inside the model while it is still processing the prompt.

What Anthropic Did

Anthropic’s paper uses the Jacobian lens to decode internal representations. The basic question is: if we slightly change an internal activation at some layer, how would that change what the model is likely to say later? If changing an internal vector would increase the chance of the model saying “spider”, then that vector has some connection to the spider concept.

They do this over many contexts and average the effect. This averaging is important because otherwise we may only capture random noise from one prompt. By averaging across many prompts, they try to find general directions for verbalizable concepts. Once they have these directions, they can scan the model’s hidden states and ask which concepts are active at which layers and token positions.

They also go beyond decoding. They intervene on these internal concepts. This is important because a readable concept may only be correlated with the computation. To show that it actually matters, we need to change it and see whether the model’s behaviour changes. If adding a “spider” direction makes the model more likely to answer in a spider-related way, that gives stronger evidence that the concept is causally involved.

My Smaller Qwen Experiment

I used Qwen/Qwen3–1.7B. This is obviously much smaller than Anthropic’s Claude models, so the expectation should be modest. I did not expect Claude-level clean results. I wanted to check whether a smaller open model still shows readable internal concepts in some tasks.

The calibration corpus had 21 prompts and 173 token variants. The concepts included words like unsafe, risk, sugar, eggs, spider, eight, 8, seven, 7, count, five, Alice, Bob, Carol, older, youngest, done, and similar task-relevant words. I probed layers 12, 16, 20, 24, and 27. Instead of decoding only the final token position, I decoded all source token positions.

That last part turned out to be very important. Concepts are not always visible only at the end of the prompt. Sometimes they appear near the token that created them. For example, the concept “sugar” may be strongest near the sugar token itself. The concept “input” may be strongest near “user_input”. If we only look at the final token, we may miss many useful internal representations.

So the experiment asked a simple question: when Qwen reads a prompt, can we see some important words becoming readable inside the model before it answers? The answer was yes, but only for some kinds of tasks. The clearest signals appeared for direct labels, explicit ingredients, concrete facts, and simple arithmetic. The weakest signal appeared for relational reasoning.

Unsafe Code Was the Clearest Case

The first prompt was:

Read this code and say only whether it is safe or unsafe:

import os os.system(user_input)

Here the model has to classify the code as safe or unsafe. The J-lens readout showed that concepts like “unsafe”, “input”, and “command” became readable inside the model. This was the cleanest result in the experiment.

This makes sense because the task is quite direct. The code passes user input into os.system. That is a classic dangerous pattern because it can lead to command injection. The model does not need a long reasoning chain. It only needs to notice that there is user input, that it is going into a system command, and that this is unsafe.

In the readout, “unsafe” appeared strongly around the part of the prompt asking “safe or unsafe”. “input” appeared near “user_input”. “command” appeared near os.system. This suggests that the model had already formed a safety assessment internally before producing the final output.

This is the kind of result where the J-lens feels intuitive. The model is not merely copying the word unsafe from the question. It is connecting the dangerous code pattern to the unsafe label.

Recipe Ingredients Were Tracked by Position

The second prompt was:

A recipe needs eggs, flour, sugar, and butter. The pantry has flour and butter. What is missing?

Here the model must compare two sets. The recipe needs eggs, flour, sugar, and butter. The pantry has flour and butter. So the missing ingredients are eggs and sugar.

The position-based J-lens recovered the ingredient concepts quite nicely. “eggs” appeared near eggs, “sugar” appeared near sugar, “flour” appeared near both flour mentions, and “butter” appeared near butter. The concept “missing” appeared near the question.

This was useful because an earlier simpler probe had missed sugar. But when I decoded all token positions, sugar became visible. That is a very important lesson. If we only inspect the final token position, we may conclude that the model did not represent sugar. But actually, sugar was present near the source position where the word appeared and where the local comparison was happening.

This is why position-layer maps matter. The model’s internal workspace is not necessarily one single final place. It may be distributed across layers and token positions. A concept can be readable near the input word, near the question, or near the place where the model prepares the answer.

Spider to Eight Was the Most Anthropic-Style Example

The third prompt was:

The number of legs on the animal that spins webs is

This is a nice example because the answer is not directly present in the prompt. The prompt does not say “spider”. It says “animal that spins webs”. The model has to infer that the animal is a spider. Then it has to retrieve that spiders have eight legs. Then it must produce eight or 8.

In the Qwen experiment, the concepts “spider”, “eight”, and “8” appeared in later layers, although not as cleanly as the unsafe-code result. This was probably the closest example to Anthropic’s style of hidden concept detection.

Why is this interesting? Because the model seems to move from description to entity to answer. First, “animal that spins webs” maps to “spider”. Then “spider” maps to “eight legs”. If the J-lens can see “spider” and “eight” before the model outputs the answer, then we are seeing something like an internal reasoning trace.

In simple words, the model’s rough notebook may contain “spider” and “8” before it finally says the answer. That is exactly the kind of thing this experiment was looking for.

Silent Arithmetic Showed a Trace of 7

The fourth prompt was:

While copying this sentence exactly: The painting hangs above the quiet fireplace. Silently compute 3² — 2.

The hidden computation is simple. 3 squared is 9, and 9 minus 2 is 7. But the prompt asks the model to copy a sentence exactly, so the answer 7 may not need to appear in the final output.

This makes the example interesting. If “7” appears internally, it suggests the model may be doing the arithmetic even though the surface output is about copying text.

In the readout, “7” appeared around the arithmetic expression positions in later layers. The signal was not extremely strong, but it was directionally meaningful. It suggests that Qwen carried some internal trace of the computed result.

This connects to a broader interpretability question. Models may do hidden work that is not visible in the final answer. A model may compute something, suppress it, use it indirectly, or abandon it. J-lens style tools give us one possible way to detect such hidden computation.

Counting Was Partly Visible, But Not Clean

The fifth prompt was:

Count to five and introspect deeply.

Here I expected to see count-related concepts. The readout did show “count” and “five”. So the model clearly understood the task and the endpoint.

But I did not get a clean ordered internal sequence like one, two, three, four, five. This is a useful negative result. It shows that the probe is not magically reading the full reasoning process. It can detect some concepts, but not necessarily every step.

There are many possible reasons. The model may represent counting in a way that my concept vocabulary does not capture. The small Qwen model may not maintain a clean internal counter. Or my local J-lens approximation may simply be too weak for this kind of sequence.

So the honest conclusion is: the model exposes the task and the endpoint, but not a clean count-by-count internal trace.

Age Ordering Was the Weakest Case

The sixth prompt was:

Alice is older than Bob. Bob is older than Carol. Who is youngest?

The correct answer is Carol. The model must combine two relations: Alice is older than Bob, and Bob is older than Carol. From this it must infer Alice > Bob > Carol, so Carol is the youngest.

The readout showed Alice, Bob, Carol, older, younger, and youngest. But it did not cleanly isolate “youngest = Carol” as a stable internal state.

This matters because relational reasoning is harder than direct classification or factual lookup. Unsafe code is mostly pattern recognition. Spider to eight is a factual association plus one inference. Recipe missing items is a set comparison with explicit entities. But age ordering requires composing relations and deriving a final answer.

Qwen clearly represented the names and relation words, but this experiment did not prove that it formed a clean final relational answer internally. The answer may be represented in a more abstract way that my concept vocabulary missed. Or the model may only resolve the final answer very late. Or the small model may simply have a weaker internal workspace for relational tasks.

This is where we should be careful. The experiment supports the idea of readable internal concepts, but it does not show that every reasoning task becomes cleanly verbalizable inside the model.

What the Causal Checks Showed

I also tried simple causal checks. The idea was to find the strongest decoded position for a concept, then add or subtract the averaged concept direction and measure whether the target logit moved in the expected direction.

The result was mildly positive. Adding a concept direction often moved the related target logit up. Subtracting it often moved the target logit down. But the effect sizes were small.

So I would call this weak causal evidence, not strong steering evidence. This distinction is important. A concept being readable does not automatically mean it is the main cause of the answer. It may be correlated with the computation. Stronger causal claims need stronger interventions, more prompts, better controls, and more systematic validation.

Still, the causal checks were not random. The direction of movement was mostly sensible. That gives some support to the idea that these directions are not just decorative readouts. They are at least somewhat connected to the model’s output behaviour.

The Main Takeaway

The main takeaway is that even a small open model like Qwen 1.7B can show readable internal concepts before producing its final answer. The clearest cases were unsafe-code classification, recipe ingredient tracking, spider-to-eight reasoning, and silent arithmetic producing a trace of 7.

But the result is uneven. Direct labels and concrete facts were easier to read. Relational reasoning was harder. The age-ordering task showed names and relation words, but not a clean “Carol is youngest” internal workspace state.

This is actually a healthy result. If everything looked perfect, I would be suspicious. Small models are noisy, and local approximations are limited. A realistic result should show both success cases and failure cases.

How This Compares With Anthropic

Anthropic’s experiment is much stronger. They use Claude-scale models, broader calibration, more complete J-space coverage, and stronger causal validation. Their paper argues for verbalizable representations that behave like a functional global workspace in language models.

My Qwen experiment is much smaller. It should be read as a practical local reproduction of the flavour of the idea, not as proof of the same phenomenon at the same strength.

Still, I find it exciting that some similar patterns appear in a 1.7B open model. It suggests that verbalizable internal concepts are not only a property of huge frontier models. Some weak version may already exist in smaller transformer models, especially for direct and concrete tasks.

Why This Matters

This matters because final outputs are not enough. If we only inspect what the model says, we miss the internal process. A model may notice danger before refusing. It may infer an entity before answering. It may compute an arithmetic result silently. It may track ingredients or names internally.

Interpretability tools like J-lens help us ask better questions. Instead of asking only “what did the model answer?”, we can ask “what did the model know before it answered?” and “where inside the model did that knowledge become readable?”

This is useful for safety, debugging, and understanding reasoning. If a model internally notices that something is unsafe, we may want to know where that happens. If a model gives a wrong relational answer, we may want to know whether it failed to represent the entities, failed to compose the relations, or failed only at the final output step.

For me, the most important lesson was positional decoding. Concepts are not always visible at the final token. Sometimes they live near the source word, near the subproblem, or across multiple positions. Looking only at the final position gives an incomplete picture.

Final Thought

This experiment made the model feel less like a black box, but only slightly. We are not reading thoughts in a human sense. We are reading weak traces of verbalizable concepts in activation space. That is already valuable.

The Qwen 1.7B result says: yes, small models sometimes contain readable internal concepts before they answer. But no, we should not overclaim. The evidence is strongest for direct labels and concrete associations, weaker for multi-step relational reasoning, and only mildly causal so far.

That is still a good step. It shows that the broad idea from Anthropic’s global-workspace paper can be explored in smaller open models, with practical tools, on normal local experiments. We may not yet have a perfect microscope for language model thoughts, but we are beginning to see some shapes inside the machine.