The Hardest Part of Fine-Tuning Isn’t the Training

On labels, failures, and why the hardest part of NLP is never the model

I spent a week building a text classifier for r/diabetes posts. I designed the labels, collected the data, fine-tuned DistilBERT, ran the baseline, and wrote the evaluation report. And when everything was done, I had a model that achieved 86.7% accuracy, but also a zero-shot baseline that got 100%.

That gap is the most interesting thing I learned. Let me tell you what happened.

What I Was Trying to Build

The idea was simple: online health communities are full of posts that do very different things. Some people share personal stories. Some make medical claims that might be accurate or not. Some ask for help. If you’re building a moderation tool, a misinformation detector, or a peer support triage system, it would be incredibly useful to know what a post is doing before you decide what to do with it.

FULL PROJECT ARCHITECTURE

So I built TakeMeter, a classifier for r/diabetes with three labels:

• personal_experience — the post narrates something that happened to the author
• health_claim — the post makes a factual or medical assertion about diabetes
• seeking_support — the post asks the community for advice or shared experience

Simple enough on paper. Much messier in practice.

The Label Design Problem Nobody Warns You About

Before I collected a single post, I spent a full evening just designing labels. This felt like procrastination. It wasn’t.

The project spec warned that “weak labels produce models that learn nothing useful.” I read that, nodded, and thought I understood it. I did not fully understand it until I started reading actual r/diabetes posts.

Here’s a post that nearly broke my taxonomy on day one:

“I was diagnosed 3 years ago and struggled with my A1C cutting carbs completely fixed it for me. Anyone else try this?”

Is this personal_experience? It's someone's story. Is it health_claim? It asserts that low-carb fixes A1C. Is it seeking_support? It ends with a community question.

The answer I landed on: label by the primary communicative purpose, not the content. If the post ends with a question directed at the community → seeking_support. If it asserts something as generally true → health_claim. If it's narrating the author's own experience with no question and no general claim → personal_experience.

That tiebreaker rule saved me from annotation chaos. I wrote it down before labeling a single example. Every time I hit an ambiguous post, I applied the rule and moved on. Without it, I would have labeled similar posts differently on different days and trained the model on noise.

The lesson: your decision rules matter more than your label names.

Collecting 200 Posts From a Platform That Blocked Me

I ran into a problem I didn’t anticipate: Reddit locked down its API for new developers in 2023. Every scraping method I tried returned a 403 error. PRAW required developer credentials I couldn’t get. The .json endpoint was blocked.

So I collected all 200 posts manually, copying and pasting directly from the r/diabetes browser page across hot, top, and new feeds. It was slower than I wanted, but it kept me close to the data. I read every post before labeling it, which made my annotations more consistent than bulk collection would have.

Final dataset: 200 posts, 67 personal_experience, 67 health_claim, and 66 seeking_support.

What Is Fine-Tuning?

Large language models like BERT are pre-trained on massive amounts of text. They already understand English grammar, context, meaning. But they don’t know anything about your specific task.

Fine-tuning is the process of taking that pre-trained model and teaching it your specific labels using your specific data. Think of it like this:

• Pre-training = teaching someone to read and write
• Fine-tuning = teaching that same person to sort your mail

You’re not starting from scratch. You’re building on top of what already exists.

Fine-Tuning: The Part That Took 15 Minutes

This surprised me. The part I was most worried about, actually fine-tuning a transformer model, took about 15 minutes on Google Colab’s free T4 GPU.

I fine-tuned distilbert-base-uncased ( is a small, fast, and light Transformer model trained by distilling the BERT-base model. It has 40% fewer parameters than BERT bert-base-uncased and runs 60% faster while preserving over 95% of BERT’s performance) on my 140 training examples for 3 epochs with a learning rate of 2e-5. The notebook handled tokenization, the train/val/test split, and evaluation metrics automatically.

The results on my 30 example test set:

Fine-tuned DistilBERT per-class metrics with precision, recall, and F1 scores for personal_experience, health_claim, and seeking_support labels. Overall accuracy 86.7%.

Not bad at all. health_claim and personal_experience were perfect. seeking_support had some confusion. But the overall picture looked solid.

Then I ran the baseline.

DistilBERT fine-tuning pipeline from raw text to predicted label in 5 steps

The Baseline Got 100%

The zero-shot Groq baseline LLaMA 3.3–70b with no task-specific training, just my label definitions as a system prompt, achieved 100% accuracy on the same 30 test examples.

Every. Single. Post. Correct.

My first reaction was that something was wrong with my code. I added debug cells, printed raw predictions, and checked that the test posts were actually being sent to the API. Everything was fine. The baseline just worked perfectly.

Here’s why I think this happened:

My labels are functionally very distinct. Health claims use clinical vocabulary. Personal experience posts use first person past tense. Seeking support posts ends in questions. These are surface patterns that LLaMA 3.3–70b was trained on, essentially the entire internet has seen millions of times, and can detect without any task-specific training.

What this tells you about fine-tuning: a small fine-tuned model doesn’t always beat a large zero-shot model. DistilBERT has 66 million parameters. LLaMA 3.3–70b has 70 billion. For tasks where the labels map cleanly to surface language patterns, the size advantage of the large model can outweigh the task-specific advantage of fine-tuning.

Fine-tuning wins when the task requires domain-specific knowledge the large model doesn’t have, or when the labels are subtle enough that a zero-shot prompt can’t capture them. My labels were clear enough that a well written prompt was sufficient.

Where the Fine-Tuned Model Actually Failed

Confusion Matrix

Even at 86.7% accuracy, all 4 wrong predictions showed the same pattern:

True label: seeking_support Predicted label: personal_experience

Here are two of them:

“Looking for something small and discreet and not heinously expensive for a medical alert bracelet.”

No question mark. Short. Declarative. The model predicted personal_experience because there's no question mark and no narrative — it just looks like a statement. But it's clearly a request for product recommendations.

“Help me with diet pls! I have T2 diabetes and my thyroid has been high for 4 months. I don’t know what to eat and I’m always on a budget. Please help me.”

This one surprised me the most. The post literally says “Help me” and “Please help me” twice. But so much personal context surrounds the request (thyroid issues, budget constraints, cooking habits) that the model weighted the narrative content over the explicit help-seeking language.

The pattern: the model learned to weight personal context over question structure. In r/diabetes, people rarely ask bare questions. They provide their situation first, their diagnosis, their frustration, and their constraints before asking for help. The model learned that personal context = personal_experience, even when that context is set up for a request.

This is the gap between what I intended the model to learn and what it actually learned. I intended to learn communicative intent. It learned surface style.

What Fine-Tuning Actually Taught Me

The 15 minutes of training were the easy part. Everything else, designing the labels, writing the decision rules, collecting and annotating the data, and analyzing the failures, took days.

Here’s what I’d tell anyone starting their first fine-tuning project:

Spend more time on labels than you think you need to. The model can only learn what your labels actually capture. If your labels are vague, your model learns something vague. If your labels have overlapping boundaries, your model learns the overlap, not the distinction. Write your decision rules before you annotate a single example.

Read your data before you label it. I read 30–40 posts from r/diabetes before committing to my taxonomy. Those 40 posts taught me more about what my labels needed to capture than any amount of thinking in the abstract.

Wrong predictions are your most valuable data. All 4 of my errors were the same mistake. That’s not a random failure, it’s a signal. The model learned something systematic and wrong, and the wrong predictions show you exactly what it learned.

Fine-tuning a small model isn’t always better than prompting a large one. For tasks with clear, surface-level signals, a strong zero-shot prompt on a capable model may outperform a fine-tuned small model. Fine-tuning adds value when the task is subtle, domain-specific, or requires knowledge the large model doesn’t have.

What I Would Do Differently

More ambiguous examples in the training set. My errors all came from posts where personal context surrounded a request. I should have deliberately over-sampled that exact pattern during data collection posts that look like personal narratives but end in requests.

Try a larger fine-tuned model. DistilBERT has 66 million parameters. RoBERTa-base or BERT-base might handle the nuanced personal_experience vs seeking_support boundary better. The choice of architecture matters, and I defaulted to the spec's suggestion without exploring alternatives.

A fourth label for mixed posts. The hardest posts to label were ones that blend all three purposes roughly equally. A “mixed” label would allow the model to express genuine uncertainty rather than forcing a wrong prediction in cases where humans themselves disagree.

The Honest Takeaway

I set out to build a classifier that could tell the difference between what a post is doing: narrating, asserting, or asking. What I actually built was a classifier that recognizes how those three things typically sound.

That’s nothing. It’s 86.7% accurate. It handles clear-cut cases well. health_claim detection is perfect.

But the gap between “recognizes style” and “understands intent” is where all the interesting failures live. And closing that gap is what makes NLP hard, not the training pipeline, not the GPU, not the library setup.

It’s the labels. It’s always the labels.

TakeMeter was built as part of AI201 at the CODEPATH. The full code, dataset, and evaluation report are on GitHub.

If you’re building something in healthtech AI or have thoughts on patient community NLP, I’d love to connect — find me on LinkedIn or @datawithrabina on Instagram.