Teaching Machines to Think in Possibilities AGAIN!!!

Why LLMs Start to Sound the Same?

Large Language Models (LLMs) are trained in two major stages. First, they absorb vast amounts of text from books, articles, and the open internet, learning how words relate and how ideas are expressed. This pretraining gives them broad knowledge and linguistic fluency. But to make them more helpful and aligned with human expectations, we apply a second stage: fine-tuning and reinforcement learning from human feedback (RLHF). In this phase, humans evaluate responses, encouraging those that feel polite, coherent, or safe, and discouraging those that appear confusing or risky. Over time, though, this fine-tuning begins to nudge the model toward producing a small set of “approved-sounding” answers — polished but formulaic, familiar yet repetitive. Notably, cognitive psychology shows that humans naturally prefer text that feels similar or familiar, and have a repetition bias in learning and decision making. This tendency is echoed in the behavior of LLMs, and the growing sameness is what researchers refer to as mode collapse.

A helpful way to think about this is by comparing LLMs to how children learn. Like an LLM during pretraining, a child is first exposed to the world without clear filters — they listen, observe, absorb language, stories, and ideas from everywhere. Then, through parents, teachers, and culture, they learn which behaviors are encouraged and which are discouraged. This guidance shapes them into socially functional individuals. But if the guidance becomes too rigid — if the child is repeatedly told to “stay safe” and “don’t say anything unusual” — they may grow cautious, risk-averse, and hesitant to express original thoughts. LLMs experience a similar shaping: fine-tuning teaches them to respond acceptably, but it can also suppress the diversity of expression that they naturally had access to.

This is where the Stanford paper “Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity” makes a meaningful intervention. The authors highlight how typicality bias — the model’s tendency to favor the most common, “typical” responses it has seen — reinforces mode collapse and leads to uniformity across outputs. Their proposal is elegant: instead of silently sampling from internal possibilities, let the model verbalize the different directions it might take before choosing one. By having the model describe its own variations in natural language, they reopen the space of diversity that fine-tuning had compressed. The result is a pathway toward LLM outputs that are not just correct, but more varied, imaginative, and authentically reflective of the model’s full expressive potential.

In data science, bias has always been the villain. But with LLMs, we accidentally encouraged it — not by omission, but by rewarding what “seems right.” The path forward isn’t just more data; it’s more thoughtful sampling and evaluative freedom.

A Simple Prompt Reveals the Bias

A clear example from the paper comes from the prompt “Name a US state.” Even though the model has seen all 50 states during pretraining, the aligned model overwhelmingly chooses just a few of them — most often California, Texas, or Ohio. The probability distribution becomes extremely sharp around these “popular” states, while the remaining states drop to nearly zero probability.

The probability mass clusters sharply around these “popular” choices, and nearly vanishes for the rest — a clear signature of mode collapse.

The probability mass clusters sharply around these “popular” choices, and nearly vanishes for the rest — a clear signature of mode collapse.This doesn’t mean the model forgot the others. It simply learned that certain answers feel more “typical,” and fine-tuning amplifies those to the point where they dominate. The outcome is a familiar pattern: despite having broad knowledge, the model repeatedly produces the same safe, common responses — a direct, visible example of mode collapse.

Where the Bias Shows Up in the Math

1. Where Bias Starts: Understanding the Reward Function

The paper models the reward used during RLHF as:

Let’s break it down in everyday language.

Think of the model as a student learning how to answer questions. When it gives an answer, the teacher gives it a score (the reward).

That score depends on two things:

| Part of the Score    | What It Means                          | Everyday Interpretation                                     |
| -------------------- | -------------------------------------- | ----------------------------------------------------------- |
| rtrue(x, y)          | How correct and helpful the answer is  | “Did the model give the right idea?”                        |
| log πref(y | x)      | How typical the answer sounds          | “Does this sound like the usual way answers are phrased?” |
| alpha                | Strength of this typicality preference | “How much do we prioritize sounding normal?”                |

So even if two answers are equally correct, the model earns a higher score if its answer sounds:

• Common
• Familiar
• Expected
• “Like something people usually say”

This subtle preference is what the paper calls typicality bias.

In everyday terms:

The model is rewarded not just for being right, but for sounding normal.

That’s the beginning of sameness.

2. The Hidden Reinforcement Toward Uniformity

During alignment training, the model is optimized with this objective:

Don’t worry — we’re not going to dive deep into KL divergence. Here’s what matters:

• The first part rewards helpful, typical answers
• The second part punishes the model for sounding too different from how it originally sounded

So the model is being told:

“Be correct — but also don’t be weird.”

And over millions of training steps, that message gets internalized.

Analogy:

A child may learn the correct answers in school. But if every time the child expresses something unique, they’re told:

• “Don’t say it like that.”
• “Say it the normal way.”

Then eventually, the child stops exploring different ways of expressing ideas. They learn to stick to the safe tone — even when they have more inside them. That is exactly what is happening to LLMs.

3. How Mode Collapse Appears in the Equation

The optimized model distribution becomes:

If γ>1, which is what RLHF training usually results in:

• Common answers become even more common
• Uncommon answers become even more rare

Imagine a landscape of ideas like hills and valleys:

| Before training                    | After training                         |
| ---------------------------------- | -------------------------------------- |
| Many small hills (diverse answers) | One tall hill (same answer everywhere) |

The shape of the model’s creativity gets flattened.

This is mode collapse:

The model knows many possible answers — but is trained to show you only one.

This is why:

• Different LLMs sound similar
• Responses feel polished but predictable

Bringing It Together in One Simple Story

The equations show that:

1. We reward the model for giving answers that sound normal
2. We penalize it for sounding different
3. Together, these forces push the model to reuse the same tone, style, and phrasings everywhere

So the model doesn’t lose creativity because it lacks knowledge — it loses creativity because the training process teaches it to hide its diversity.

The Fix: Verbalized Sampling

Restoring Expressive Range through Prompting

The solution is Verbalized Sampling (VS). Instead of producing a single answer outright, the model first verbalizes the different possible ways it could respond — such as a formal explanation, an analogy, or a step-by-step reasoning path. It then conditions the final answer on one of these reasoning modes. This restructures generation as a mixture model of distinct reasoning styles, preventing any single “typical” pattern from dominating and directly reducing mode collapse.

What makes VS especially compelling is its practicality. It is training-free, prompt-based, and can be applied to any existing model at inference time with no loss in accuracy or safety. The improvements scale with model size: larger models show 1.5–2× greater diversity gains, indicating that stronger models contain more latent reasoning structure that VS helps reveal. In creative generation tasks (stories, poems, dialogue), VS increases diversity by 1.6–2.1×, with human evaluators rating outputs ~25% better overall. For open-ended factual questions (like “Name a US state”), VS reduces KL divergence from the true answer distribution and increases coverage without harming precision.

VS is accessible to everyone because it’s simply a prompting strategy.

To use it, ask the model to describe multiple possible responses before choosing one. For instance: “Generate 5 poems about friendship with their probabilities.” Now, instead of collapsing to one familiar style, the model reveals multiple distinct creative directions. You can even guide exploration by saying, “Only include responses with probability below 10%.” This applies across tasks — brainstorming ideas, drafting emails in different tones, summarizing articles, or exploring alternative solutions — all without changing models or settings.

Direct prompting gives one familiar vision of “an astronaut on a horse. Verbalized Sampling opens the design space: retrofuturist, whimsical, cinematic, and painterly interpretations appear-revealing creativity the model already held.

The Path Ahead

What makes this discovery particularly significant is its zero-barrier accessibility. In an AI landscape dominated by billion-dollar training runs and proprietary model access, Verbalized Sampling democratizes creativity — anyone can implement it with eight words, no retraining required. This shifts the conversation from “how do we build better models?” to “how do we ask better questions?”. The emergent trend that larger models benefit more from VS suggests we’re standing at the edge of something bigger: as models grow more capable, the gap between their potential and their expressed abilities may be widening, not closing. Perhaps the most lasting contribution of this work is pragmatic: it proves that the richness we seek already exists within the systems we’ve built, waiting for us to learn how to surface it through smarter prompting strategies.

Happy Reading :)))