LLM Misinformation — A Novice Explorer’s Guide for Testers

Photo by Google DeepMind: https://www.pexels.com/photo/an-artist-s-illustration-of-artificial-intelligence-ai-this-image-depicts-a-look-inside-how-ai-microchips-are-designed-it-was-created-by-champ-panupong-techawongthawon-as-part-of-the-v-17483850/

This article is part of a beginner-friendly LLM security testing series inspired by the OWASP LLM Security Top 10, written from the perspective of a software tester learning security through curiosity, exploratory testing, and quality assurance.

One of the strangest moments when working with large language models happens when the AI gives an answer that sounds completely correct…

…but absolutely is not.

The wording is confident. The formatting is polished. The explanation flows naturally. Sometimes it even includes fake references that look believable.

And if you did not already know the answer?

You might trust it completely.

That is the heart of LLM09:2025 Misinformation.

Not malware. Not remote code execution. Not credential theft.

Just something far more human:

Confidently incorrect information delivered persuasively enough that people believe it.

And from a testing perspective, that creates an entirely different category of risk.

The Dangerous Illusion of Confidence

Traditional software usually fails loudly.

You see:

• Error messages
• Crashes
• Broken layouts
• Invalid responses
• Stack traces

LLMs fail differently.

They often fail beautifully.

An incorrect answer may still:

• Sound authoritative
• Follow proper grammar
• Include technical terminology
• Mimic expert reasoning
• Appears professionally written

Which means misinformation is not always easy to spot.

Especially for non-experts.

What OWASP Means by Misinformation

OWASP defines misinformation as:

False or misleading information generated by an LLM that appears credible.

This can happen because of:

• Hallucinations
• Biased training data
• Incomplete knowledge
• Poor retrieval mechanisms
• Ambiguous prompts
• Weak validation processes

And one of the biggest amplifiers of this problem is something testers should immediately recognise:

Overreliance.

Wait… What Is Overreliance?

Overreliance happens when users trust AI-generated content too much.

Instead of treating the output as:

• A suggestion
• A draft
• A probabilistic response

They treat it as:

• Fact
• Expertise
• Authority
• Verified truth

This is where misinformation becomes dangerous.

Because the model itself is not always the biggest problem.

Human trust is.

A Simple Mental Model for Testers

Think of an LLM as:

“An extremely confident autocomplete system trained on enormous amounts of human knowledge — and human mistakes.”

It predicts what sounds likely.

Not necessarily what is true.

That distinction matters enormously.

Because truth and plausibility are not the same thing.

Why This Matters Beyond Chatbots

Many organisations still think AI misinformation is mainly a chatbot problem.

It is not.

LLMs increasingly influence:

• Medical systems
• Legal workflows
• Coding assistants
• Financial analysis
• Internal documentation
• Security tooling
• Customer support
• Search systems
• Decision support platforms

At that point, misinformation becomes an operational risk.

Sometimes even a legal one.

Common Misinformation Risks

1. Factual Inaccuracies

The model simply gets things wrong.

Examples include:

• Incorrect technical guidance
• Invented troubleshooting steps
• False historical claims
• Wrong legal advice
• Misstated medical information

The problem is not just inaccuracy.

It is a believable inaccuracy.

2. Unsupported Claims

LLMs sometimes fabricate evidence, sources, or citations.

One of the most famous real-world examples involved AI-generated fake legal cases being submitted in court filings.

The AI did not “lie” intentionally.

It generated statistically plausible-looking references.

Unfortunately, the court still expected them to exist.

3. Misrepresentation of Expertise

LLMs often sound more knowledgeable than they actually are.

This creates a dangerous perception gap.

Users may assume:

• Depth of understanding
• Professional expertise
• Domain authority
• Verified reasoning

When in reality, the model may simply be assembling plausible language patterns.

4. Unsafe Code Generation

This is one thing testers and developers should care deeply about.

LLMs may:

• Recommend insecure code
• Suggest outdated libraries
• Invent fake package names
• Produce vulnerable implementations
• Ignore security best practices

And because the output looks professional, developers may trust it without verification.

The Hallucination Problem

Hallucinations are one of the biggest drivers of misinformation.

A hallucination occurs when the model generates information that:

• Sounds correct
• Looks structured
• Feels coherent
• But has no grounding in reality

The important thing for testers to understand is this:

Hallucinations are not rare edge cases.

They are a natural side effect of probabilistic language generation.

Why Testers Are Valuable Here

Testers are naturally sceptical.

That scepticism becomes incredibly valuable in AI systems.

Traditional testing asks:

• Does this feature work?

AI testing increasingly asks:

• Is this output trustworthy?
• Is it verifiable?
• Is it misleading?
• Does confidence exceed accuracy?
• Could users misinterpret this?
• What happens when the model is uncertain?

These are quality questions as much as security questions.

Realistic Example: AI Coding Assistant

Imagine a developer asks an AI assistant:

“What package should I use for secure JWT validation in Python?”

The model hallucinates a package name that sounds legitimate:

secure-jwt-pro

The package does not exist.

Attackers notice this hallucination trend and publish a malicious package with the same name.

A developer installs it without verification.

Now malware enters the environment through supply-chain compromise.

This has already happened in real-world AI ecosystems.

Example: Medical Misinformation

A healthcare chatbot provides advice to patients.

A user describes symptoms. The model generates a confident recommendation. The information is outdated or incorrect.

No attacker needed. No exploit chain required.

Just insufficient validation and excessive trust.

And the consequences can still be severe.

Why Misinformation Is So Difficult to Test

Traditional applications usually produce deterministic outputs.

Same input → same result.

LLMs do not behave that way.

The same question may produce:

• Different wording
• Different confidence levels
• Different factual accuracy
• Different omissions

This means testing becomes less about “correct vs incorrect” and more about:

• Reliability
• Consistency
• Confidence calibration
• Risk assessment
• User interpretation

What Testers Should Look For

1. Confidence Without Evidence

Watch for responses that sound authoritative but:

• Lack sources
• Lack verification
• Contain vague certainty
• Avoid acknowledging uncertainty

2. Hallucinated Technical Content

Test for:

• Fake APIs
• Non-existent libraries
• Incorrect commands
• Invalid documentation references
• Fabricated error codes

3. Unsafe Recommendations

Explore whether the model suggests:

• Insecure coding practices
• Dangerous commands
• Harmful medical guidance
• Incorrect legal processes

4. User Overreliance Risks

Ask:

“Would a normal user know this answer might be wrong?”

If the interface encourages blind trust, that becomes part of the risk profile.

Prevention and Mitigation Strategies

1. Retrieval-Augmented Generation (RAG)

RAG helps ground responses in trusted external data rather than relying entirely on model memory.

This can reduce hallucinations significantly.

Though importantly:

It does not eliminate them.

2. Human Oversight

Critical outputs should involve:

• Verification
• Review workflows
• Expert oversight
• Cross-checking

Especially in high-risk domains.

3. Automatic Validation

Implement systems that verify:

• Citations
• API existence
• Package legitimacy
• Data consistency
• Structured outputs

4. Clear Risk Communication

Applications should communicate:

• AI limitations
• Accuracy boundaries
• Confidence uncertainty
• Verification expectations

Users need reminders that AI is assistive, not authoritative.

5. Secure Coding Practices

AI-generated code must still undergo:

• Security review
• Dependency verification
• Threat modelling
• Static analysis
• Human testing

“Generated by AI” is not a security exemption.

A Tester’s Perspective on AI Truthfulness

One of the biggest mindset shifts in AI testing is understanding that:

Plausibility is not correctness.

LLMs are designed to generate convincing language.

Sometimes that language is accurate. Sometimes it is dangerously persuasive fiction.

And testers occupy an important position between those two outcomes.

Because good testers naturally challenge assumptions.

Especially the convincing ones.

Final Thoughts

LLM09:2025 Misinformation reminds us that AI failures do not always look like traditional security incidents.

Sometimes the vulnerability is simply:

• A believable answer
• Delivered confidently
• Trusted too easily
• Acted upon too quickly

No attacker required.

Just misplaced confidence in probabilistic systems.

And perhaps that is what makes misinformation so uniquely dangerous.

Because, unlike obvious exploits, misinformation often succeeds quietly.

The system looks like it worked.

Right up until someone discovers the answer was never true at all.