Teaching Machines to Read: How We Turn Words Into Numbers

You just wrote a scathing movie review:

“This movie was absolutely terrible.”

You hit submit.

But the AI doesn’t flinch. It doesn’t feel your anger. It doesn’t even understand the words. Because no matter how intelligent a model seems, at its core it’s just a math machine — and math doesn’t speak human language. It only understands numbers.

This is the very first, often overlooked step in every natural language processing (NLP) project: turning raw text into numbers. Without this bridge, no AI can read, understand, or respond to human language.

Welcome to the art and science of text vectorization. Let’s walk through this journey — from the most basic methods to smarter ones — and see how we taught machines to read.

Why Is This Harder Than It Sounds?

Language is messy, ambiguous, and ever-changing:

• Words have multiple meanings (“bank” can mean a riverbank or a financial institution)
• New slang and trends appear every week
• Word order completely flips meaning (“The cat sat on the mat” vs “The mat sat on the cat”)
• Most documents use only a tiny fraction of possible words

So how do we compress all this complexity into clean numerical vectors? Here are the classic techniques every NLP practitioner should master.

Essential NLP Terms You’ll Keep Seeing

• Corpus: Your entire dataset of text (e.g., 50,000 movie reviews)
• Document: One single piece of text (a review, email, tweet, or article)
• Vocabulary: All unique words across your corpus
• Token: A single cleaned word (after lowercasing, removing punctuation, etc.)

1. One-Hot Encoding: The Naive Starting Point

Imagine a tiny vocabulary: [cat, sat, mat, dog, ran]

Each word becomes a vector as long as the vocabulary, with 1 in its position and 0s everywhere else:

• cat → [1, 0, 0, 0, 0]
• sat → [0, 1, 0, 0, 0]

Pros: Simple, intuitive, and unambiguous. Cons:

• Extremely sparse (mostly zeros)
• Zero semantic meaning (“cat” is equally different from “dog” as from “democracy”)
• Breaks with any new word (Out-of-Vocabulary problem)

Great for learning the concept. Terrible for real production use.

2. Bag of Words (BoW): Counting What Matters

Bag of Words throws away word order and simply counts word frequencies.

Example: Doc 1: “The cat sat on the mat” → [2, 1, 1, 1, 1] Doc 2: “The cat sat on the cat” → [2, 2, 1, 1, 0]

Why it works surprisingly well: Spam emails contain “free”, “win”, and “click” more often. Positive reviews are rich in “great”, “loved”, and “amazing”. Frequency carries real predictive power.

Limitations: Completely ignores order and context. “The dog bit the man” and “The man bit the dog” look identical.

Still one of the strongest and fastest baselines for text classification.

3. N-Grams: Adding Local Context

Instead of single words, we count sequences of words.

Sentence: “The food was not good”

• Bigrams: “the food”, “food was”, “was not”, “not good”

Now the model sees “not good” as one unit instead of two conflicting signals.

Strength: Excellent at capturing phrases like “highly recommend”, “not bad”, or “customer service”. Trade-off: Vocabulary size explodes, increasing sparsity and computational cost.

Best practice: Combine unigrams + bigrams and keep only the most frequent ones.

4. TF-IDF: Weighing Word Importance

TF-IDF improves on Bag of Words by answering: Is this word actually important?

TF-IDF = Term Frequency × Inverse Document Frequency

• TF: How frequent is the word in this document?
• IDF: How rare is the word across the entire corpus?

Common words like “the” and “is” get near-zero scores. Rare, distinctive words like “backpropagation” or “transformer” get high scores.

Best used for: Search engines, document similarity, and information retrieval. It powered traditional search for years and remains a reliable, interpretable technique.

5. Custom Features: Engineering Intelligence by Hand

Sometimes the best features come from human insight, not automation.

Examples for spam detection:

• Contains ALL CAPS words? (1/0)
• Number of exclamation marks
• Contains URLs or suspicious links?
• Ratio of special characters

Examples for sentiment analysis:

• Count of positive/negative words from a lexicon
• Presence of negation (“not”, “never”, “hardly”)
• Use of intensifiers (“absolutely”, “extremely”)

Advantages: Highly interpretable, lightweight, and often outperform ML features on small datasets. Disadvantage: Requires domain expertise and manual work.

Final Thoughts: Choosing the Right Approach

• One-Hot Encoding → Learning only
• Bag of Words → Fast baseline for classification
• N-Grams → When phrases matter
• TF-IDF → Best for search and similarity
• Custom Features → When you need explainability and domain knowledge

All these traditional techniques share one critical limitation: they treat words as isolated symbols. They understand frequency and statistics, but not meaning, synonyms, or real-world knowledge.

That’s why the next evolution — word embeddings (Word2Vec, GloVe) and transformer models (BERT, GPT) — changed everything.

But that’s a story for the next article.