From Amazon Reviews to Numbers: A Hands-On Tour of One-Hot, Bag of Words, and TF-IDF

How I took 128 real Amazon product reviews and turned them into features a machine-learning model can actually chew on — and what I learned about where these classical techniques still shine in 2026.

Why bother with “classical” text features at all?

If you have been anywhere near an LLM in the last two years, you have probably heard that “embeddings solved text.” They did — for a lot of problems. But if you are building:

• a spam filter with 100k labelled examples,
• a BM25-powered search box (Elasticsearch, OpenSearch, Lucene),
• a cold-start classifier for a brand-new product line, or
• a compliance-audited system where “why did the model fire?” needs a human-readable answer,

…then Bag of Words (BoW) and TF-IDF are still in the toolbox. They are fast, deterministic, interpretable, and an honest baseline you should always beat before reaching for a neural model.

This post walks through a small, self-contained project that does the whole pipeline on real, scraped Amazon reviews. The full source is in amazon_text_features/ next to this file.

Step 1 — Get real data (not toy sentences)

Every blog post on TF-IDF uses the same three cooked-up sentences about cats and dogs. I wanted the messiness of real user-generated content, so I wrote a tiny BeautifulSoup scraper that grabs the 3–13 reviews Amazon shows directly on each product detail page (/dp/<ASIN>) and loops across ~20 popular ASINs.

Two practical notes if you want to reproduce this:

1. Set a real User-Agent header. Without one Amazon returns a stripped-down page and you will extract exactly zero reviews.
2. Anchor your selector correctly. The top reviews on /dp pages are rendered inside celwidget blocks, not the div[data-hook="review"] wrapper you see on the dedicated reviews page. Select [data-hook="review-body"] and walk up to the nearest celwidget to grab the star rating.

After about 30 seconds of polite scraping I had 128 reviews across 14 products — Echo Dots, AirPods Pro, Kindles, an Apple Watch, a Ninja blender, a PS5 controller, a Nespresso machine, and so on. A few ended up being in Spanish and Arabic, which is a lovely reminder that real data never matches the shape your slides promised.

Step 2 — Clean the text (the boring part that matters most)

A review like "I LOVE it!!! Sound is 🔥. Read more" is not something a counting-based model can work with. Every step in the cleaning pipeline kills a specific kind of noise:

stepkillswhylowercaseLOVE vs loveavoids vocabulary duplicatesdrop Read moreAmazon truncation markerotherwise it becomes one of the most frequent tokensstrip punctuation / digits!!!, $199they rarely help classical modelstokenize—gives you units to countremove stopwordsthe, and, isthey appear in every document → no signallemmatizespeakers → speakertightens the vocabulary

After processing, my 128 reviews reduced to 11,138 tokens spanning a 3,461-word vocabulary. The top words were exactly the product-review clichés you would expect — use, one, like, great, noise, sound, quality — which is a nice sanity check.

Step 3 — Three ways to turn text into numbers

The assignment asks for three encodings. Here is how I think about them:

One-Hot Encoding (document level). For each review, build a binary vector over the whole vocabulary: 1 if the word appears, 0 otherwise. It is the simplest thing that works and the easiest to explain to a non-technical stakeholder. Its fatal weakness is that it throws away how often something is said: a review mentioning "amazing" once and another mentioning it ten times look identical.

Bag of Words (CountVectorizer). Same vector shape, but store the actual counts. Now a review that hammers on "sound" three times will rank differently from one that drops the word once. BoW is frequency- aware but still order-blind: "not good, very bad" and "good, not very bad" hash to nearly the same vector.

TF-IDF (TfidfVectorizer). The trick that makes BoW actually useful. Take the BoW count and divide it by how common the word is across the whole corpus. Words like good or use, which appear almost everywhere, get pushed toward zero. Rare but distinctive words like cancellation, haptic, or hardwire stay loud. Mathematically:

tfidf(t,d)=tf(t,d)⋅log⁡N1+df(t)tfidf(t,d)=tf(t,d)⋅log1+df(t)N​

In my corpus, the highest-IDF words were exactly the long-tail product features that appeared in just one review. The lowest-IDF words were the generic review vocabulary. That is the whole story of TF-IDF in one experiment.

Step 4 — The “aha” moment: look at one review under all three

The fastest way to feel the difference is to encode the same review three times and print the top-weighted tokens.

• OHE just lists every unique word in the review. No ranking.
• BoW surfaces the most repeated words — which are almost always filler (one, like, use).
• TF-IDF surfaces the words that this review says and few others do. That is exactly what a downstream classifier wants to see.

Once you have seen this side-by-side even once, you stop reaching for plain BoW unless you have a very specific reason (Naive Bayes is one — its underlying math prefers raw counts).

Step 5 — Sparsity, the thing nobody warns you about

Every one of my three matrices came out ~98.15 % zero. That is normal — reviews are short, vocabularies are long, and most words do not appear in most documents. But it has two huge practical implications:

1. Never store these dense. A 1-million-document × 200k-vocab corpus is a 200-billion-cell matrix. It has to live in CSR or equivalent compressed form or you run out of RAM before you start training.
2. Classical pipelines do not scale forever. Once you are in the tens-of-millions-of-documents range, even sparse storage becomes painful, and that is one of the reasons industry moved to dense embedding pipelines for web-scale retrieval and clustering.

Step 6 — A mini sentiment classifier (and a lesson about class imbalance)

For the use-case I treated 4–5 star reviews as positive, 1–2 star as negative, and dropped the neutral 3-star reviews. Then I trained two models on each feature set:

• Logistic Regression with class_weight="balanced"
• Multinomial Naive Bayes

The headline accuracy looks great — ~97 % on the test split. But that is almost entirely because the test split contains 31 positives and 1 negative. The interesting metric is recall on the negative class, and with only five one-star reviews in the whole corpus, no model is going to learn that cleanly. This is a realistic constraint: Amazon surfaces highly-rated reviews first, so any pipeline that scrapes the top-of-page reviews will inherit the same lopsided distribution.

Lessons from the small experiment:

• TF-IDF gives logistic regression a small, consistent edge by silencing filler words.
• Naive Bayes prefers raw BoW counts. Rescaling with IDF can actually hurt it — its probability estimates are count-based.
• Never trust a single accuracy number on imbalanced data. Always print per-class precision/recall.

Step 7 — Where these techniques break (and where they still win)

The textbook weakness of BoW/TF-IDF is semantics. They are lexical, not conceptual:

• "audio is excellent" vs "sound is great" → zero shared tokens → zero similarity. Obviously wrong.
• "battery lasts forever" vs "battery dies quickly" → almost all tokens shared → high similarity. Also obviously wrong.

That is why dense embeddings (Word2Vec, GloVe, BERT, modern sentence-transformers, and the embedding endpoints from model vendors) took over semantic search and clustering. They map synonyms close and antonyms far, which lexical methods cannot do.

But BoW/TF-IDF still win when you need:

• Speed and predictability. A TF-IDF + linear classifier can train on millions of documents in minutes on a single laptop.
• Interpretability. Each feature is a word you can literally point at. Try that with a 1024-dim embedding.
• Exact-match retrieval. BM25 (TF-IDF’s cousin) still beats many embedding systems for keyword queries — especially when the user types an identifier, product code, or rare name.
• Cold starts. No training data? Just fit a TF-IDF and rank by cosine similarity. It is not state-of-the-art, but it works on day one with zero labels.

Takeaways

1. Preprocessing is 80 % of the game. Before you touch any encoder, make sure you understand exactly what “a token” means in your corpus.
2. Always look at a single document’s top features. It is the fastest way to develop intuition about what your encoding is actually rewarding.
3. Watch sparsity and class imbalance. Both will bite you long before modelling choices do.
4. Know why you would pick the classical tool. If your answer is only “because it is in every tutorial”, you should probably be using an embedding model. If your answer is “because I need interpretability and speed”, BoW/TF-IDF are still excellent.

What is in this folder

amazon_text_features/
├── scrape_amazon.py               # the scraper
├── amazon_reviews.csv             # 128 real reviews
├── build_notebook.py              # programmatic source of the notebook
├── Text_Feature_Engineering.ipynb # the executed notebook
├── report.md                      # 1-2 page observations report
└── blog.md                        # this post

To reproduce everything from scratch:

python scrape_amazon.py        # writes amazon_reviews.csv
python build_notebook.py       # writes Text_Feature_Engineering.ipynb
jupyter nbconvert --to notebook --execute Text_Feature_Engineering.ipynb \
    --output Text_Feature_Engineering.ipynb

That is the whole pipeline — from raw HTML to a small sentiment classifier — with every step explained in plain English. If you are learning NLP in 2026, this is still the best way to build the intuition you will need when you graduate to embedding-based models: know exactly what you are replacing, and why.

Source code — https://github.com/vijaygokarn130/ml-classic-concepts