Why LLMs Still Struggle with Simple Tasks — And Why It Matters

Large Language Models (LLMs) like ChatGPT, Claude, and Gemini can summarize research, draft content, and analyze complex information. Yet they still stumble on something an elementary school student can do: count letters in a word.

OpenAI researchers highlighted this in a 2025 paper Why language models hallucinate | OpenAI, using a simple prompt:

“How many D’s are in DEEPSEEK?”

DeepSeek‑V3 answered 2 or 3 across ten tests. Meta AI and Claude 3.7 produced answers as high as 6 and 7. A human can instantly see the correct answer: 1.

This is an example of an intrinsic hallucination — the model contradicts the prompt itself. Counting letters isn’t a complex task, so why do advanced models fail?

The Core Issue: Probabilistic Models Doing Deterministic Work

LLMs don’t “count.” They predict the next most likely token based on patterns in their training data. And because there aren’t millions of documents explicitly stating how many times a letter appears in a word, the model has no reliable pattern to anchor to. The result: hallucinations.

Letter counting is a simple task, there is only one answer and the answer does not change. It is a deterministic task, not a probabilistic task.

Have Models Improved? Yes… But Not Fully

I was curious if the models had improved since the release of the paper. I repeated the same experiment with Claude, Gemini, and ChatGPT. All three models initially produced incorrect answers, then corrected themselves.

This raises an interesting question: If the model can self‑correct, why doesn’t it perform that check before responding?

It suggests that multiple models may have been retrained with similar guardrails — leading to the same pattern of “wrong first, right second.”

More Tests

I tried a slightly more complex example:

It failed to identify both ‘p’s in ‘paper”. However, when I prompted it again, it was able to provide the correct answer.

The problem isn’t the letter counting. The problem is trust.

If an LLM can fail at a simple task, you cannot assume correctness for more complex tasks without verification.

Why This Matters

LLMs are expensive to run — financially and environmentally. As the industry continues to adopt AI at scale, we need to be intentional about which tasks require probabilistic models and which should be handled by deterministic programs.

LLMs are incredible for:

• reasoning over messy, unstructured text
• generating content
• synthesizing insights
• guiding users through complex workflows

But for tasks that require guaranteed correctness — like counting, validation, parsing, or rule‑based logic — we should rely on deterministic methods.

And if needed, use the LLM to generate the deterministic code, not to perform the deterministic task itself.

For example, if letter counting was truly a task that you needed to complete, instead of asking an LLM to do it, ask it to provide you the code to do it. It would give you something like this:

These few lines of code would guarantee you the correct output every time.

The Takeaway

As AI systems advance, the challenge isn’t just making them smarter — it’s knowing when not to use them. Distinguishing probabilistic vs. deterministic tasks will help us build AI workflows that are more reliable, more efficient, and more sustainable.