The First Edition of the Newspeak Dictionary

In George Orwell’s 1984. the Party has just produced an Eleventh Edition of Newspeak, a dictionary of words in which “every concept that can ever be needed, will be expressed by exactly one word, with its meaning rigidly defined and all its subsidiary meanings rubbed out and forgotten.”

By changing the language, “the whole climate of thought will be different. In fact there will be no thought, as we understand it now. Orthodoxy means not thinking — not needing to think. Orthodoxy is unconsciousness.”

Maybe it’s the computer science degree, but reading that makes me think about AI. I use AI to clean up my emails, to help me through a coding bug, to not need to think…

Of course, language is always changing. There was a big upset ten years ago, when dozens of nature words were cut from the Oxford Junior Dictionary and replaced with more computer-related words: acorn for attachment, buttercup for blog, clover for chatroom.

But the way that AI has been changing the words, paired with some people’s growing dependency on it and its spotlight in mainstream media seems almost dangerous. The output of unconsciousness is a homogenization of language.

The fear is, for me: hermeneutical injustice — when someone can suffer an injustice by being deprived of the words or knowledge needed to communicate their own experience. Some literature has already begun viewing AI through this lens. When language tilts, certain experiences become harder to name.

If we start all sounding the same, using the same AI generated text, then it seems like we are on track with Orwell’s dystopian vision: “Every year fewer and fewer words, and the range of consciousness always a little smaller.”

Maybe we’re ready to write the very first edition of the Newspeak Dictionary.

AI Speak

Words generated in excess. https://arxiv.org/pdf/2406.07016

The words generated by AI has already changed the landscape of our language. AI tends to use certain words more often than people do.

One word kept coming up everywhere: delve. Its usage spiked dramatically after the release of ChatGPT.

The same was true in academic literature. Scroll to the end for a list of sources that cover this trend.

Researchers have catalogued the wider phenomenon. Delve belongs to a whole vocabulary of words appearing at elevated rates in scientific writing, business communication, online discourse.

These AI-flavored words are bleeding into spoken language. People are even starting to talk like ChatGPT.

How do I visualize words?

At the start of the project I met with the artist Kyle McDonald to talk through the idea. He brought up his project *Unlearning Language, in which people in a room communicate without a machine that has been trained to interpret their language and gestures. It is an experiment in returning to “when humans were uninterpretable.” That stayed with me. What does it mean to make AI strange to itself* again?

I started by reading about “lost” words. I read Robert Macfarlane’s Landmarks (a nice review here). Words like coachan, “a slender stream hidden by the overgrown vegetation, so that the stream cannot see out of its own bed,” and daal’mist, “mist which gathers in valleys overnight and is exhaled when the sun rises,” were some of my favorites.

I wondered if there was a way to visualize the words generatively. I worked through some LoRA experiments, fine-tuning Stable Diffusion on small image sets to teach the model concepts I could then prompt with. I followed Oscar Keyes’ *Blooming From Noise workshop materials, which build on Eryk Salvaggio’s [Flowers Blooming Backward Into Noise](https://www.youtube.com/watch?v=zNA7sPm-zlQ)* (2023). I trained on flowers first to get familiar with the pipeline.

Love, Peace & Joy

LoRAs work well to generate images in a particular style. The words I’d selected from Landmarks, while beautiful, didn’t have a specific visual style. The same problem existed for my curated list of AI words; there wasn’t a single ground-truth aesthetic. What does delve look like?

Back to the drawing board

I went back to staring at the list of AI words. They seemed to describe abstract thought; connecting, examining, exploring, synthesizing. But I noticed that many of them were nature-related, or had a distinct usage tied to the natural world that had fallen out of common use.

Delve in AI contexts means “to make a careful or detailed search for information,” as in delved into the past. But the Merriam-Webster entry also gives: cave, hollow.

Confluence in AI prose means a coming together of factors. It also means the flowing together of two or more streams, a meeting of rivers.

Culmination in AI prose is the high point of an effort. In astronomy, it’s the precise moment when a celestial object passes through its highest point in the sky.

This pattern held across many of the words. I wanted to bring back their literal natural meanings, the ones AI prose has drifted from.

I wondered: was the AI still using these words in their literal natural meanings, somewhere in its representations? Or had the meanings been hollowed all the way out?

Looking inside the model’s head

To answer this, I needed a way to peek inside a language model’s representations. The technique I used is called a Sparse Autoencoder (SAE), which is an active area of mechanistic interpretability research.

I’d describe it as a de-tangling.

When a language model reads a word, that word becomes a long list of numbers. For Gemma-2–2B, the model I used, that vector has 2,304 entries at each layer. This list is called an activation, the model’s internal representation of the word in this context. The same word in a different context produces a different activation.

The model represents many concepts in tangled, overlapping superposition, usually packing more concepts than it has dimensions by representing them as overlapping directions in activation space. You can’t point to entry #142 and say “that’s the river neuron.”

A Sparse Autoencoder is a small neural network you train on top of the language model, whose job is to *untangle* these activations into separate, interpretable concepts.

If I use a prism metaphor: the model’s activation is white light, a tangled mix of colors. The SAE, like a prism, separates it into individual colors (we call them features).

Technically, the SAE projects the 2,304-dimensional activation up to a much higher dimension (16,384 features for the SAE I used) but enforces that almost all of those entries must be zero for any given word. Only a handful are allowed to fire, forcing each of the dimensions to mean something specific .

After training, researchers can examine each feature and see what kinds of text make it activate. For Gemma specifically, Google DeepMind released GemmaScope, which has SAEs trained on every layer of the model. You can play with it yourself on Neuronpedia.

I chose to feed the model the same word in two contexts, and look at which features fire in each.

• An AI-prose context: “The confluence of these multifaceted factors represents a transformative paradigm shift.”
• A nature-writing context: “At the confluence of the two gentle rivers, the clear waters mix into a wide, rippling flow.”

The SAE tells me which features fire on “confluence” in each. The difference between them shows what’s present in one context that’s absent in the other.

The Process

For each word and each context:

1. Tokenize the prompt.
2. Locate the token span covering the target word (since a word like confluence may split across multiple sub-tokens — typically conf + luence in Gemma’s SentencePiece tokenizer).
3. Run a forward pass with a hook on model.model.layers[20] that captures the residual stream output.
4. Pass that activation through the SAE’s encoder to get the 16,384-dimensional feature vector.
5. Aggregate across the word’s sub-tokens (mean across positions; I tried sum, max, and last as alternatives — mean was most stable).

Then I subtract: concrete_features – hollow_features. Features with the largest positive values are the candidates for "what AI prose strips away," the features that fire when the word is used physically, but not when it's used abstractly.

I started with two words: delve and confluence.

The first sanity check was on GPT-2 small with Joseph Bloom’s residual stream SAE. The diagnostic for delve surfaced one really interesting concrete-context feature whose Neuronpedia auto-interpretation read “specific geographic locations and their fauna.” That was exactly the kind of feature I was hoping would activate. But for confluence, the diagnostic came back muddled: the top concrete features were about “time-related phrases,” “tech conferences,” “being on the brink.” With GPT-2 small, the literal meaning of confluence wasn’t clearly separable in this SAE. This might mean that the “river” meaning had been so rare in training data that the model never built clean features for it.

I turned to Gemma-2–2B + GemmaScope. The feature space is much larger and the auto-interpretations are cleaner. For confluence, the top concrete-context feature came back as “references to rivers and their geographic features.” For delve, “references to archaeological activities and discoveries.” That was close enough for my purposes.

Steering: making the model speak as the river

Next, I wanted to use the “natural” feature to make the model behave differently.

This process is called feature steering, and it works by inserting a hook between layers that modifies the activation as it flows forward. Instead of just observing what a feature represents, you add a scaled version of that feature’s decoder direction to the residual stream, and let the model continue forward as if that concept were strongly present.

modified_activation = activation + strength × sae.W_dec[feature_idx]

I played with it first on Nueropedia to understand how it works. Turning up the steering controls made the feature more and prevalent. But, it was finicky and could break if the strength was up too high.

An example of steering on Nueronpedia

For confluence, with the rivers feature steered at strength 4 on the prompt “The confluence of”, Gemma’s baseline output (no steering) was about the S&P 500, pure financial expository AI prose. With steering, the same prompt, same model, same seed produced text about the Brahmaputra and the Meghna, the Ganges-Brahmaputra-Meghna delta, the largest river delta in the world.

I realized that the river meaning was already inside the model; it had just been buried beneath more frequent abstract uses. The steering didn’t create the rivers. It surfaced them!

With the SAE steering as scaffolding, then added an engineered “persona” to shift the chat dialogue to first-person. The persona produces the voice, and the steering grounds it.

Here’s a table of the word paired with the persona:

For each word in the dictionary, the process is:

1. Run the SAE diagnostic with hand-crafted hollow and concrete prompts.
2. Inspect the top concrete-context features on Neuronpedia, pick the one whose auto-interpretation matches the literal meaning.
3. Write a persona prompt that asks the model to be the thing, like a river or a cave.
4. Tune the steering strength (typically 6–10) until the output is in-character but coherent.
5. Add to the JSON.

I have the process written in a colab notebook.

Once the chat was built, I wanted to make it visually interesting and interactive. The user starts by opening a dictionary. When they select a word, the screen will glitch from the AI use of the word into the natural definition, and they’ll enter a new space.

Here, a video plays, with a warping effect that deforms in real time based on mouse position. The video surface stretches and squeezes toward the cursor while the edges stay pinned. The video feed uploads each frame as a WebGL texture, and a fragment shader adds UV distortion, pixel smearing (multi-sample blending along the stretch direction), and chromatic aberration to sell the feeling of physical material being pulled. Then, the text gets drawn to an offscreen Canvas 2D element and uploaded as a second texture into the same render pass, so it warps identically to the video underneath it.

The experience moves through three phases. On load, a title and subtitle float over the warping video. Then, the user is dropped into a hunt phase, where a single hotspot is placed at a randomized position each session. The hotspot has its own slow influence on the mesh: a pulsing upwelling in the vertex shader, plus concentric brightness ripples and spiraled UV sampling in the fragment shader. When the mouse enters the hotspot radius, a glyph surfaces under the cursor and the water begins to open. Clicking triggers a glitch burst that transitions into a terminal-style chat overlay.

The First Edition of the Newspeak Dictionary

Each entry shows a word and its sleek AI-prose definition. I got these definitions straight from asking ChatGPT.

When you click an entry, the typography glitches.

The definition warps and reforms into the meaning in a nature-related context. I used a variety of different dictionaries to find definitions I liked the most.

The user must dig (delve) through the pixels to uncover a portal. The screen will glow, and the mouse will be bigger.

When found, a small chat field invites: “ask the river…” Each chatbot is tailored to the natural meaning.

It takes a little bit of time for the chatbot to respond on the first query, since it has to “warm up.” After the first one, it’s pretty quick to reply. You only get three questions per chat — the point isn’t a long AI conversation.

Access the live deployed version.

What the project is, and isn’t

My intention with the project isn’t exactly a critique of AI. I just want to document the way these particular words have drifted into a managerial / academic register, but that the natural meanings are still encoded inside the model, recoverable through the right combination of prompt and intervention.

The cure for hollowed language is literary attention. The model knows how rivers are written about because writers wrote about rivers. We need to ask the model to remember.

I hope to add more words and build the dictionary out more fully. I’ll also be looking at other better ways to visualize words. I’m not sure the SAE / prompt persona approach was the best way to do it. I also want to add something for the common phrases that AI text often does, like “It’s not X, it’s Y.” I also would like to figure out a way to make a tribute to the nature words I read in Landmarks. There’s something so beautiful about that list.

Tools, references, sources

SAE / interpretability tooling:

• sae_lens — Python library for loading pretrained SAEs and running them on top of HuggingFace transformer models.
• **GemmaScope** — Google DeepMind’s open SAE release for the Gemma-2 family.
• **Neuronpedia** — public dashboard of SAE features with auto-interpretations, max-activating examples, and a test-input box.
• Anthropic’s *Towards Monosemanticity* a foundational SAE paper.
• Adam Karvonen’s intuition guide to SAEs.
• The example feature-steering interface for Gemma-2–9b-it on Neuronpedia, which shaped the chatbot’s UX.

Model & deployment:

• google/gemma-2-2b — the base pretrained model.
• bitsandbytes 4-bit NF4 quantization for fitting Gemma on a T4.
• Modal Labs — serverless GPU hosting; auto-scales to zero so the chatbot only costs money when actively used.

Conceptual / literary:

• George Orwell, *1984.*
• Robert Macfarlane, *Landmarks*.
• The Oxford Junior Dictionary controversy of 2015 (*The Guardian*).
• Generative hermeneutical injustice: Recent papers extending Miranda Fricker’s framework to AI.

Empirical work on AI vocabulary:

• Multiple arxiv papers measuring word-frequency shifts in academic abstracts pre/post-ChatGPT (arxiv.org/pdf/2403.07183, arxiv.org/pdf/2403.16887, arxiv.org/pdf/2406.07016).
• Studies showing AI-flavored vocabulary leaking into spoken language (arxiv.org/abs/2409.01754, *Scientific American*).
• Curated lists: gptzero.me/ai-vocabulary, github.com/nanxstats/llm-cliches, the Reddit threads cataloguing the words.

Conversations and inspirations:

• Kyle McDonald pointing me toward *Unlearning Language.*
• Eryk Salvaggio, Flowers Blooming Backward Into Noise (2023).
• Oscar Keyes, *Blooming From Noise* workshop materials.

View the deployment at hollow-words.vercel.app.

AI Use Statement

Given the subject of this project, it would be strange not to disclose how AI was involved in making it.

I worked through much of this project in conversation with Claude, including drafting and iterating the diagnostic notebooks (the multi-word SAE pipeline, the steering ablations); helping write the implementation spec for the website backend; and helping create the glitch effects on the frontend.

The technical decisions, the curatorial choices about which words to include, the prompts for the SAE diagnostics, the analysis of what the SAE features showed, the project’s framing and conclusions are mine. The infrastructure work (Modal deployment, Next.js frontend, Tailwind components) was implemented in Claude Code.

The Gemma-2–2B model itself, of course, is a Google open-weights model, and the GemmaScope SAE is from Google DeepMind’s open release.

I want to note that the outputs of the chatbot in the dictionary — the river speaking, the cave answering — are generated live by Gemma at runtime, with the persona prompts and steering applied as described above.

A project critiquing AI’s effect on language while using AI to make the project is a tension I want to leave visible rather than smooth over. It wouldn’t work without AI.