I Built a Sentiment Classifier and Caught Myself Cheating

How I built an NLP project, found a 40% performance illusion in my own work, and what it taught me about being honest with metrics

I’m an Electrical and Electronics Engineering student at Marmara University, but for the past few months I’ve been pivoting hard into Data Science. The plan was simple: build a portfolio that proves I can do the work. Three projects, four weeks, then start applying to junior DS roles.

What I Set Out to Build

The plan was a “Review Intelligence System” — an end-to-end NLP pipeline that takes Amazon reviews and does three things:

1. Sentiment classification — is this review positive, negative, or neutral?
2. Fake review detection — is this review suspicious?
3. Aspect-based sentiment — which parts of the product is the customer talking about?

I was going to use the Amazon Reviews 2023 dataset from Hugging Face, train both classical ML models and DistilBERT, deploy it as a Streamlit app, and call it a day.

That was the plan. What actually happened was more interesting.

The Easy Part: Sentiment Classification

I pulled 99,000 reviews across three categories — Electronics, Books, and Beauty, 33K each. The first thing I noticed was that 62.8% of reviews were 5-star. Another 17.3% were 4-star. The 1-star and 2-star reviews combined? Less than 12%.

This is the thing nobody tells you when you start: real-world data is brutally imbalanced. If I just trained a classifier to predict “5 stars” for everything, I’d be right 63% of the time without learning anything.

So I made the first real decision of the project: instead of predicting 5 classes (1–5 stars), I’d predict 3 — negative (1–2 stars), neutral (3 stars), positive (4–5 stars). Even after this, the dataset was 79% positive, 12% negative, 9% neutral. Still imbalanced, but workable.

Building Baselines I Couldn’t Skip

Before I touched anything fancy, I wanted to know what “stupid” looks like. I trained a dummy classifier that just predicts “positive” for everything. It hit 79% accuracy.

This is the moment that taught me the most important lesson of the project: accuracy lies on imbalanced data. The dummy classifier looks 79% right, but its macro F1 — the average F1 across all three classes — is 0.29. It’s literally getting zero signal on negative and neutral reviews. Just memorizing the majority class.

So I made macro F1 my primary metric. Then I built two real baselines:

• TF-IDF + Logistic Regression — bag of words with a linear classifier. Macro F1: 0.66
• TF-IDF + LightGBM — bag of words with gradient boosting. Macro F1: 0.62

Wait, what? LightGBM was supposed to be better than logistic regression. Everyone says gradient boosting beats linear models. Why did LogReg win by 4 F1 points?

I dug into it and found something I’d never read about in tutorials: TF-IDF features are 99.79% sparse. In a 50,000-dimensional vector, only about 100 values are non-zero. Tree-based models split on individual features, and in such sparse spaces, the splits are essentially random. Linear models, which combine all features simultaneously, handle sparsity gracefully.

This was the first “this is not what I expected” moment. The expected hierarchy of models didn’t apply here. I spent 30 minutes thinking I was doing something wrong before I realized: no, this is just how high-dimensional sparse text features work.

DistilBERT: The Big Gun

I had a free Google Colab account and a Tesla T4 GPU available, so I figured I’d fine-tune DistilBERT next. DistilBERT is a smaller, faster version of BERT — 40% fewer parameters, 60% faster inference, but it still captures most of BERT’s performance.

The training took 22.8 minutes. Three epochs with class weighting (negative class got 2.81x weight, neutral 3.67x, positive 0.42x — I had to fight the imbalance somehow).

Here’s where things got interesting again.

The accuracy of DistilBERT was lower than my logistic regression baseline. 0.847 vs 0.852. Not by much, but lower.

For about ten minutes, I was confused. Did I do something wrong? Did I burn 22 minutes of GPU time for a worse model?

Then I looked at the macro F1–0.70 vs 0.66. The macro F1 went up while accuracy went down.

The model became more balanced across classes. It got slightly worse at predicting “positive” (because it was no longer over-predicting it) but dramatically better at predicting “neutral” (recall went from 34% to 60% — a 76% relative improvement).

This is when sentiment classification really clicked for me. Accuracy and macro F1 measure different things. Accuracy rewards majority-class bias. Macro F1 punishes it. On imbalanced data, you want the second one.

The contextual embeddings did exactly what they’re supposed to do: they understood that “good but slow shipping” is mixed sentiment, not just positive. Bag-of-words couldn’t make that distinction. BERT could.

| Model | Macro F1 | Neutral F1 | Inference | | Dummy | 0.29 | 0.00 | < 0.01 ms | | LogReg | 0.66 | 0.37 | < 0.01 ms | | LightGBM | 0.62 | 0.28 | 0.02 ms | | DistilBERT | 0.70 | 0.44 | 2.59 ms (GPU) |

I felt good. I had three working models, a clear winner, and metrics that actually meant something.

Then I made the mistake that became the real story of this project.

The Fake Review Detector That Worked Too Well

Module B was supposed to be fake review detection. The problem was that I didn’t have ground-truth fake labels — Amazon doesn’t tell you “this review is fake.” Real-world fake detection always has this problem.

So I used weak supervision. I wrote heuristic rules:

• If the review is unverified AND has an extreme rating (1 or 5 stars) → probably fake
• If the text is a generic phrase like “Good”, “Great”, “Five Stars” AND under 5 words → probably fake
• If a single user has more than 10 reviews → probably promotional

I summed these rules into a “fake score” and labeled anything above a threshold as fake. About 21% of my reviews got flagged.

Then I trained a LightGBM classifier on this. I gave it the text (TF-IDF) plus the metadata features: rating, helpful_vote, word_count, verified_purchase. I added the 4,666 confirmed text duplicates from my EDA as additional fake examples.

The result?

93.4% F1 score. ROC-AUC of 0.994. PR-AUC of 0.98.

I was thrilled for about five minutes.

Then I looked at the unsupervised baseline I’d also trained — Isolation Forest, which doesn’t use any labels. It had F1 of 0.30.

A 63-point gap between supervised and unsupervised on the same task. That’s… not normal.

The Moment I Caught Myself

I almost didn’t catch it. I was about to write up the results, push to GitHub, and call Module B done. The numbers looked great. Why dig deeper?

Because the gap was bothering me. Real-world hard tasks don’t usually show a 63-point F1 gap between supervised and unsupervised. If unsupervised is at 30%, the task is hard. If supervised is at 93%, the task isn’t hard. Both can’t be true at once.

So I looked at what features the model was using. Top three by importance:

1. word_count_clean
2. rating
3. verified_purchase_int

These were the same features I used to define the labels.

The model wasn’t detecting fake reviews. It was reverse-engineering my heuristic rules. It learned: “if rating is extreme and verified is false, predict fake.” Of course it did — I’d basically given it the answer key.

This is label leakage, and it’s one of the most insidious bugs in machine learning because the model still produces correct predictions on the test set. The metrics look great. The model is “right” — it’s just learning the wrong thing.

I couldn’t ship this. Not because the F1 was wrong, but because the F1 didn’t measure what I thought it measured. The model wasn’t generalizing to detect fake patterns; it was reproducing my own labels.

Rebuilding It Honestly

I rebuilt the entire fake detector with three changes:

1. User-stratified train/test split. In my naive version, the same user could appear in both train and test. If a “prolific user” had 30 reviews, the model could memorize that user’s pattern from training and apply it at test time. With GroupShuffleSplit on user_id, no user appears in both sets. Verified: zero overlap.

2. Text-only features. No more rating, verified_purchase, or word_count as inputs. The model could only see the text itself. If it wanted to predict fake, it had to learn from the language, not from the metadata that defined the labels.

3. Stricter labels. I raised the threshold from 3 to 4, requiring stronger evidence before calling something fake. This dropped the labeled fake rate from 21% to 16% — closer to the industry estimate of 5–15%.

I retrained. The results:

F1: 0.56.

Not 0.93. Not 0.85. Just 0.56.

The 40% drop between V1 and V2 wasn’t a bug or a regression. It was the size of the leakage problem. My naive model wasn’t 93% good at fake detection — it was 56% good at fake detection plus 37% good at memorizing my own rules.

Why F1 = 0.56 Is Actually Fine

This was the part I had to convince myself of. After seeing 93%, anything in the 50s feels like failure.

But look at it this way:

• A random baseline on this dataset (predicting fake at the 20% base rate) would get F1 around 0.30
• My model gets 0.56 — an 87% relative improvement over random
• More importantly: Precision@Top100 is 0.77

That last metric matters. In production, a Trust & Safety team can’t manually review every flagged review. They can review maybe 100 a day. If they take the top 100 most suspicious from my model, 77 of them are actual flagged reviews. That’s actionable. That’s deployable. That’s useful.

F1 of 0.56 sounds bad. “Of the top 100 suspicious reviews, 77 are real fakes” sounds great. They’re describing the same model. The framing changes everything.

What I Actually Learned

I’m going to be honest: I almost shipped the 93% version. The temptation was real. The numbers looked great. Nobody was going to check my work in detail. I could have padded my resume and moved on.

What stopped me was a single instinct: when something looks too good, look harder. The 63-point gap between supervised and unsupervised was the smoking gun. I just had to be willing to actually investigate it instead of celebrating.

A few things crystallized for me through this:

Sanity checks aren’t optional. Every project should have a baseline that’s obviously dumb (always-positive classifier) and a baseline that doesn’t use the labels (unsupervised approach). Both are diagnostic tools. The dummy classifier proved my real models were learning something. The Isolation Forest proved my supervised model was cheating.

Features defining labels can’t be features in the model. This sounds obvious in retrospect. It wasn’t obvious when I was building V1. The temptation to “use all available features” is strong, especially when they correlate with the target. But correlation with a label you defined isn’t the same as correlation with the underlying truth.

Production metrics > academic metrics. F1 looks bad at 0.56. Precision@Top100 looks great at 0.77. Both describe the same model. When you’re talking to non-technical stakeholders or thinking about deployment, you need metrics that map to actual decisions. “Of the top 100 suspicious reviews, X are real” is something a product manager understands. F1-macro is not.

Be willing to commit your mistakes. I have two versions of the fake detector in my GitHub repo. V1 (naive) and V2 (corrected). I didn’t delete V1. The comparison between them — the 40% drop — is now the most valuable part of the project. It shows I can build something, evaluate it skeptically, and fix it. That’s a more interesting story than “I built a 93% F1 classifier.”

What’s in the Repo

The code is on GitHub: https://github.com/3RAV0/review-intelligence-system

It has six notebooks (EDA, preprocessing, baselines, DistilBERT fine-tuning, fake detection V1, fake detection V2), three findings documents totaling about 2,000 lines of methodological discussion, and all the visualizations.

The README has the full results tables, architecture overview, reproduction instructions, and a “lessons learned” section that’s basically an expanded version of this article.

What’s Next

This was Project 1. I’m working on two moreprojects coming:

• Customer Churn + CLV Prediction Suite — XGBoost, SHAP explainability, BG/NBD model for customer lifetime value, and behavioral segmentation. This will be more classical tabular ML — a different skill set than NLP.
• Hybrid Recommendation System — Collaborative filtering, content-based filtering, and a two-tower neural network for the actual cold-start case. This is the area I’m most excited about because recommendation systems are everywhere but rarely show up in junior portfolios.

Both should be faster than this one was, because the infrastructure (preprocessing modules, evaluation framework, project structure conventions) is now in place. And because, honestly, I’ll know to look for the leakage from the start this time.

Final Thought

If I’d started this project six months ago, I would have shipped V1 of the fake detector and put 93% F1 on my resume. I would have felt good about it. I might even have gotten away with it in interviews — most interviewers don’t dig into your evaluation methodology in detail.

But I would have known. And eventually, in some real job, the same kind of self-deception would have caused real damage. Better to catch myself now, on a portfolio project, than later when someone’s relying on the output.

Honest 0.56 F1 with a quantified leakage analysis beats fake 0.93 F1 every time.

• If you found this useful, the GitHub repo has more detail than this article — including the three deep-dive findings documents that go into the methodology in detail. Feel free to reach out: bahceci.mehmet@outlook.com

Mehmet Bahçeci