LLM Hacking: A Practical Guide to Safe Data Annotation in Research.

Avoid systematic error and validate your LLM-annotated findings with these evidence-based strategies

Your Latest Study Might Be Wrong. And you probably don’t know it yet.

You used LLM(GPT, Claude, Gemini, Deepseek, LLama etc) for data annotation. You ran your analysis. Your p-value came back significant: p = 0.04, Yaayyy!!! Write. Publish. Done.

A few months later, someone tries to replicate this work. Same data. Different LLM or slightly Different prompt. Statistically insignificant: p = 0.52. Ooops!!!

Same data. Same hypothesis. Same LLM. One word difference in the prompt. The result? Significant finding becomes NOT significant.

This problem is slowly becoming a standard crisis in research.

New research by Bauman et al (2025) analyzed 13 million labels from 18 different LLMs across 2,361 hypotheses, and their findings show that

You can make almost any hypothesis “statistically significant” by using any LLM configuration choices

They coined this phenomenon “LLM hacking”: systematic bias introduced by how researchers implement LLMs, leading to faulty conclusions

When these false findings influence policy, medicine, or resource allocation, people get hurt.

There are evidence-based practices that stop/mitigate this.

I’ve distilled the 92-page evidence-based safety recommendations proposed in Baumann et al, 2025' s work into a practical pocket guide showing you:

Part 1: How to avoid LLM hacking (concrete strategies)

Part 2: How to document your work (so reviewers trust it)

Part 3: How to validate your findings (making sure they’re real)

Part 4: Detailed Checklist to use before Submitting your LLM-based Experiment for Peer Review.

If you use LLMs for research, this is for you…

PART 1: How to Avoid LLM Hacking

Here are the 6 concrete strategies:

The 6 steps to avoid LLM hacking. Each step reduces risk. All 6 together produce research that can be trusted.

Here are the 6 concrete strategies:

1. Use Large(r) Models

**Research shows that, on average, most state-of-the-art (SOTA) LLM models we use have a one-in-three chance of LLM Hacking. However, Models with ~70B parameters have a 31% hacking risk, compared to Models with 1–2B parameters, which have a 50% hacking risk. That’s a 20% difference.**

Practical recommendation: Use the largest available model for your annotation task. (GPT-4, Claude 3 Opus, etc.)

Caveat: Larger models help but aren’t magic. Expect diminishing returns as model size increases.

Scaling relationships for LLM hacking risk and annotation performance. Bauman et al. 2025

2. When in doubt, use Few-Shot Prompting instead…

Findings show that detailed prompts with examples dramatically reduce LLM variance and hacking risk compared to zero-shot (when you provide no examples to the LLM and it only relies on its internal knowledge).

Practical Recommendation:

• Include a few examples in the prompt. The number of a few-shot examples is not strict but depends on use cases. It is often advisable to provide a variety of examples relative to the use cases in your specific dataset.
• Be explicit about what you want: Research shows that a detailed description in a prompt provides significant stability and this lower risks of LLM hacking compared to brief/incomplete instructions.
• Document your exact prompt and other variations (you’ll need it later)

Caveat: Even good prompts vary and may not produce the same outcomes every time. You still need validation, and this counts as an acceptable limitation to your research work.

3. Test Multiple Configurations

If you only test a single LLM on one prompt, you’ll miss the variance. As such, different configurations on the same data can produce dramatically different results (like p = 0.04 vs p = 0.52).

Never rely on a single LLM configuration for hypothesis testing.

Practical recommendation:

• Test at least 2–3 different models
• Try 2–3 prompt variations
• Report results across ALL configurations, including p-values and effect sizes, not just your “best” result

This prevents p-hacking via LLM configuration choices.

4. Always Pre-Register Your Choices and document them

You’ve run your analysis. Once you see your results (say: p-value is 0.07). Not quite significant.

So you think: “Let me just try a different LLM temperature… maybe different few-shot examples… adjust the prompt slightly… just to be thorough.”

You try again: p = 0.04. Significant?! But you’ve just p-hacked your way to a false positive.

Practical recommendations:

• Before running the analysis, lock everything in writing: Which model was used? Which prompt? What configuration? Which analysis plan?
• Write it down (even a simple document)

5. Use Prompt Engineering Tools

Manual prompt testing is inefficient.

You’ve written 10 different prompts. You’ve tested them manually. You’re getting inconsistent results. Some work great on your validation set but fail on actual data.

Then there is the issue of changing LLMs. Now you have to reconstruct the same prompts using the new LLM configuration Specs…

Practical recommendations:

-Use tools like DSPY to systematically adjust and test prompt variations.

-Document which version you use, and Report performance across variations

DSPY is Perfect for avoiding manual prompt engineering

Want a full deep dive

💡 Want to automate your prompt optimization and reclaim 10 hours per project? I’ve written a detailed guide on DSPY implementation that walks you through the entire optimization process with code examples. Check it out here → https://medium.com/@KNew_Mikel/stop-writing-prompts-manually-treat-them-like-code-instead-24e3ab938b57

6. Human Expert Annotation is Key

Human expert annotation plays a crucial role in mitigating LLM hacking risks.

Critical caveat: High human agreement rates do NOT predict low LLM hacking risk. Meaning: If humans agreed 95% on a task (Perfect agreement), the LLM might still introduce systematic bias in your analysis. Moreover, High human agreement does not predict low LLM hacking risk

Practical Recommendation:

• Always validate LLM performance on YOUR specific task or downstream analysis, regardless of human agreement rates or other general metrics transfer.
• Use models and prompts with high annotation performance for hypothesis testing. When F1 scores fall below 0.5, LLM hacking risk typically exceeds 40%. Even when LLM-derived p-values are far from 0.05, error rates can still be quite high and substantial.

This figure reveals how proximity to the p = 0.05 threshold creates fundamental instability in research conclusions. Emphasizing the importance of Ground truth expert human annotation in LLM-based findings

PART 2: How to Document Your Work

Document Everything. No Exceptions. Reviewers can’t spot LLM hacking if you don’t show your work. Transparency is the best defense.

What reviewers need to see(Use GitHub Anonymous for initial submission to preserve anonymity):

PART 3: How to Validate your findings and mitigate unintentional LLM hacking Risks (The Most Important Part)

To mitigate unintentional LLM hacking Risk, it is important to address the issue of LLM annotation bias or unreliable LLM outputs. The 2 main approaches used in the paper for this purpose include:

Design-based supervised Learning(DSL) —introduced by Egami et al (2025). DSL is a bias-correction technique for LLM annotation that combines LLM predictions with human annotations to estimate and correct for systematic bias in the LLM’s outputs, producing pseudo-labels and outcomes that are more reliable than LLM-only predictions.

Confidence-Driven Inference(CDI) (Gligorić et al, 2025): CDI is another bias-correction technique that uses the LLM’s confidence scorings together with the human annotations. Using the LLM predictions based on their confidence levels allows it to effectively reduce the impact of low-confidence, potentially biased predictions.

To that effect, the type of error we should consider mitigating, depending on the use cases and research priorities, includes: Type I Error, aka False positive (claiming something is significant(true) when it’s not). Most LLMs make this type of error- False findings look real often through Hallucination

Type II Error, aka False negative (Something that is significant (True) but misclassified as False)

Practical recommendations:

The figure compares some Model selection strategies across ground truth sample sizes and mitigation techniques, emphasizing the importance of bias correction methods based on research priorities. Source Bauman et al. 2025

1- Always Use Human Annotations to Validate: Findings show that even 100 human annotations outperform pure LLM approaches for catching false positives.

Practical recommendations:

A- Gold Standard:

• Collect about 1,000 human annotations and use them as your main ground truth; generally, outperform any LLM-based approach in terms of LLM hacking risk on average.
• If necessary, let the LLM supplement for the rest of the annotations required for the study.
• *Can help significantly lower false positive risk, establishing robust and high confidence in findings

B — In case of Moderate Resource:

• Collect 100 human annotations and use statistical correction techniques such as Confidence Driven Inference(CDI) and Design-based supervised Learning(DSL) on your annotations.
• Report trade-offs. This can help catch fewer true positives and reduce LLM hacking risk significantly more than using LLM-only annotations.

C — In case of Limited Resources:

• Use a strong LLM with more parameters
• Few-shot prompting
• However, these findings can only be reported as preliminary and need further analysis and validation.
• *Better than nothing, but risky. Be honest about limitations.

2. Choose Your Validation Approach Based on Your Research Type

This table shows which validation strategy to use depending on your research goals and error priorities

What This Means in Practice: Is your research

a) Claiming Novel Discovery?

• A false positive could mislead the field. So it is recommended to use: Maximum human annotations (1,000+) OR pure human labeling only. In this case, the best way to truly control Type I error with LLMs is to minimize LLM use

b) Claiming Replication Study?

• You’re verifying someone else’s work. Missing real effects is more harmful (False negative) than false positives. SO it is recommended to use a Mix of human annotations and LLM with statistical correction techniques to mitigate LLM Hacking risks. In this scenario, LLM use for the annotation task is more liberal while still maintaining some level of control

c) Exploratory Work?

• You’re generating hypotheses, not making final claims. Then maybe focus on using a Strong, large LLM (70B+ parameters) for the annotation task, and ~100 human-annotated samples for validation. This helps balance both error types while remaining practical. However, this can’t be the main reported findings proposed.

THE BOTTOM LINE

LLM annotation is powerful. But without safeguards, it introduces systematic bias that looks like real findings.

This matters because we’re making critical decisions based on these “findings.”

A false finding about hate speech detection gets published. A platform implements it. The system fails. Real people get hurt.

A false finding about misinformation detection leads to over-moderation. Legitimate speech gets suppressed.

A false finding shapes policy. Resources get wasted. The field moves in the wrong direction.

And nobody catches it. [DANGER AHEAD]

Part 4 — QUICK CHECKLIST before Submitting your LLM-based Experiment for Peer Review.

QUICK CHECKLIST Before you submit your LLM-annotated study:

 [] Used the largest available model (70B+) whenever possible
 [] Used few-shot prompting (with examples)
 [] Test and report multiple models and prompt variations
 [] Make and provide your pre-registered analysis plan
 [] Report results across ALL configurations
 [] Document exact prompts and model versions used and tested
 [] Collect human annotations for validation (or explained why not)
 [] Report Type I / Type II error trade-offs
 [] Released code and data (or explained limitations)
 [] Included a 
    [] transparency and 
    [] ethical consideration statement about LLM annotation methods

You're ready to submit.
Best of Luck : )

Passionate about AI for social good and content moderation. Building safer digital spaces, one model at a time. Let’s connect on LinkedIn.

I share insights on AI/LLM safety, content moderation, and Practical ML.