Text Vectorization in NLP

Before feeding text into machine learning models, we need to convert raw text into numerical form — this is where text vectorization comes in.

In this post, I’ve shared how different vectorization methods like One-Hot Encoding, Count Vectorizer, N-Grams, and TF-IDF transform documents into vectors using Python.

Key Terminologies

• Document: A single piece of text (e.g., one article, one review).
• Corpus: A collection of documents.
• Feature: A measurable property of a document (usually words or tokens).

Tools Used

• Python 🐍
• scikit-learn
• NLTK / spaCy
• Matplotlib (for visualization)

Brainstorming

I started with one question:

• How can I convert words like “apple” or “I love NLP” into something a model can understand?

Here’s the journey I followed:

• One Hot Encoding
• Count Vectorization
• Bag of Words (BoW)
• N-Grams
• TF-IDF

Example Document

1. One Hot Encoding

Basic representation — either a word exists (1) or not (0) Explanation: It assigns a unique number to every word and marks 1 where it appears.

Output Vector Table

When to Use:

• Great for small vocabularies.
• Not ideal when vocabulary is large (sparse vectors, high memory).

2. Count Vectorizer

Counts the number of times each word appears. Explanation: The vector for each document contains word frequencies.

Output Vector Table

When to Use:

• Useful for frequency-based models like Naive Bayes.
• Doesn’t capture meaning or context beyond frequency.

3. Bag of Words

This is just a concept — CountVectorizer implements it.

• Ignores grammar and word order
• Focuses only on word counts

4.. N-Grams

Captures context by grouping n words together (bi-gram, tri-gram)

Explanation: Captures short phrases like “machine learning” instead of treating words in isolation.

Output Vector Table

When to Use:

• Good when word context matters (e.g., “not good” vs “good”).
• Feature space grows fast with large n.

5. TF-IDF Vectorizer

Balances word frequency with how rare the word is across documents.

Explanation:

• TF (Term Frequency): How often a word occurs in a doc
• IDF (Inverse Document Frequency): How rare a word is across all docs

TF-IDF reduces the weight of common words and highlights informative ones.

Output Vector Table

When to Use:

• When you need weighting based on importance.
• Common in search engines, text classification.

Other Text Vectorization Techniques

• Hashing Vectorizer
• Word2Vec
• GloVe
• FastText
• ELMo
• BERT embeddings

NLP #MachineLearning #TextVectorization #TFIDF #BagOfWords #Python #DataScience #AI #100DaysOfCode #scikitLearn #spacy #NLTK #WordEmbeddings #MLAlgorithms #NaturalLanguageProcessing #TechCommunity #LinkedInLearning #Visualization