Bag-of-Words

The Simplest Representation

The bag-of-words (BoW) model represents a document as a vector of word counts. The process:

1. Build a vocabulary from all unique words across all documents in the corpus.
2. For each document, create a vector with one entry per vocabulary word.
3. Each entry records how many times that word appears in the document (term frequency) or simply whether it appears (binary).

Consider three sentences:

• “climate change affects oceans”
• “ocean temperatures are rising”
• “climate policy changes slowly”

The vocabulary is: {affects, are, change, changes, climate, ocean, oceans, policy, rising, slowly, temperatures}. Each sentence becomes a vector of length 11. Most entries are zero.

BoW with scikit-learn

from sklearn.feature_extraction.text import CountVectorizer

texts = [
    "climate change affects oceans",
    "ocean temperatures are rising",
    "climate policy changes slowly"
]

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(texts)

print(f"Shape: {bow_matrix.shape}")           # (3, 11)
print(f"Vocabulary: {vectorizer.get_feature_names_out()}")
print(f"Dense matrix:\n{bow_matrix.toarray()}")

CountVectorizer handles tokenization, lowercasing, and vocabulary construction automatically. The result is a sparse matrix – most entries are zero, so scikit-learn stores only the nonzero values to save memory.

Limitations of Bag-of-Words

BoW has fundamental problems that no amount of tuning can fix:

1. No word order. “Dog bites man” and “man bites dog” produce identical BoW vectors. The representation discards all sequential structure.
2. No semantics. The words “car” and “automobile” are treated as completely unrelated — they occupy different dimensions of the vector with no connection between them.
3. Huge, sparse vectors. A corpus with 50,000 unique words produces vectors of length 50,000, almost entirely zeros. This is computationally wasteful and makes similarity comparisons unreliable in high-dimensional space (the “curse of dimensionality”).
4. Common words dominate. Words like “the”, “is”, and “and” appear in almost every document, inflating their counts without carrying useful information.

BoW is a starting point, not a destination. Its simplicity makes it useful as a baseline, and its speed makes it practical for very large corpora where more sophisticated methods are too slow.