How Does Google Know ‘Apple’ is a Company and Not a Fruit?

Have you ever wondered How Google Knows that ‘Apple’ is a Company and Not a Fruit? Or how your phone’s keyboard predicts the next word so accurately? Or how Siri understands whether you’re asking a question or giving a command?

The answer lies in a fascinating field of Artificial Intelligence called Natural Language Processing (NLP) and more specifically, in a technique called Token Classification.

Let’s break it all down, step by step.

— — —

🌍 First Things First: What is NLP?

Natural Language Processing (NLP) is the branch of AI that helps computers understand, interpret, and generate human language.

Think of it this way: computers are great at numbers and logic. But human language? That’s messy, ambiguous, and full of context. “I saw the man with the telescope”. Did you use the telescope, or did the man have it? Humans figure this out from context. Teaching machines to do the same is what NLP is all about.

Where do we see NLP every day?

• 🤖 Chatbots — When you chat with customer support bots, NLP helps them understand your question
• 🔍 Search Engines — Google doesn’t just match keywords; it understands the meaning of your query
• 📧 Email Filters — Gmail detecting spam uses NLP to understand email content
• 🎙️ Voice Assistants — Alexa, Siri, and Google Assistant all rely heavily on NLP

— — —

What is Token Classification?

Before we understand token classification, let’s understand what a token is.

A token is simply a unit of text — usually a word, but sometimes a character or sub-word. When we break a sentence into tokens, we call it tokenization.

Sentence : "John loves playing football in Paris"
Tokens   : ["John", "loves", "playing", "football", "in", "Paris"]

Token Classification means assigning a label to each individual token in a sentence. Instead of understanding the whole sentence at once, we analyze it word by word.

Why does token-level understanding matter?

Because meaning lives in details. Consider:

”Apple released a new iPhone in California”

• “Apple” here is a company, not a fruit
• “California” is a location
• “released” is a verb/action
• “new” is an adjective

Without token-level understanding, a machine would treat this sentence as just a bag of words. With it, the machine extracts structured, meaningful information — just like a human would.

The three most important token classification tasks are:

1. Named Entity Recognition (NER)
2. Part-of-Speech (POS) Tagging
3. Chunking (Phrase Detection)

Let’s explore each one! 🚀

— — —

1. Named Entity Recognition (NER) — “What kind of thing is this word?”

What is NER?

Named Entity Recognition is the task of identifying and classifying named entities in text, things like people, organizations, locations, dates, and more.

Think of it like highlighting important nouns in a text and putting them in categories.

Real-world example:

“Elon Musk Founded SpaceX in 2002 in California”

| Token      | Entity Label|
|--------------------------|
| Elon Musk  | PERSON      |
|------------|-------------|
| SpaceX     | ORGANIZATION|
|------------|-------------|
| 2002       | DATE        |
|------------|-------------|
| California | LOCATION    |
|------------|-------------|

The machine reads each word and asks: “IS this a person? A place? A company A date?”

What is BIO Tagging?

NER uses a special labeling format called BIO tagging:

• B- = Beginning of an entity
• I- = Inside an entity (continuation)
• O = Outside (not an entity)

Why do we need this? Because some entities spam multiple words!

New   → B-LOC   (Beginning of location)
York  → I-LOC   (Inside the same location)
City  → I-LOC   (Still inside the same location)
is    → O       (Not an entity)
a     → O
wonderful → O
place → O

Without BIO tagging, the model wouldn’t know that “New”, “York”, and “City” together form one entity.

Applications of NER:

• 📰 News analysis — extracting key people and events from articles
• 🏥 Medical records — identifying drug names, diseases, and patient info
• 💼 Resume parsing — extracting candidate names, companies, and skills
• 🔍 Search engines — understanding who or what a query is about

Quick Code Demo (spaCy):

import spacy

# Load English model
nlp = spacy.load("en_core_web_sm")

text = "Elon Musk founded SpaceX in 2002 in California."
doc  = nlp(text)

print("Named Entities:")
for ent in doc.ents:
    print(f"  {ent.text:<20} → {ent.label_}")

Output:

Named Entities:
  Elon Musk            → PERSON
  SpaceX               → ORG
  2002                 → DATE
  California           → GPE

2. Part-of-Speech (POS) Tagging

What is POS Tagging?

Part-of-Speech Tagging assigns each word its grammatical role — noun, verb, adjective, adverb, preposition, and so on.

Why is POS Tagging important?

It helps resolve word ambiguity. “Book a flight” (verb) vs “Read a book” (noun) same word, different roles!

It is the foundation for more complex NLP tasks like chunking and parsing

Used in grammar checkers, text-to-speech systems, and machine translation

Quick Code Demo (spaCy):

import spacy

nlp  = spacy.load("en_core_web_sm")
text = "The clever fox quickly jumped over the fence."
doc  = nlp(text)

print(f"{'Token':<15} {'POS Tag':<10} {'Description'}")
print("-" * 40)
for token in doc:
    print(f"{token.text:<15} {token.pos_:<10} {token.tag_}")

Output:

Token           POS Tag    Description
----------------------------------------
The             DET        DT
clever          ADJ        JJ
fox             NOUN       NN
quickly         ADV        RB
jumped          VERB       VBD
over            ADP        IN
the             DET        DT
fence           NOUN       NN

3. Chunking (Phrase Detection) — “Which words belong together?”

What is Chunking?

Chunking (also called Shallow Parsing) groups individual tokens into meaningful phrases. While POS tagging tells us the role of each word, chunking tells us which words form a group.

A “chunk” is a phrase — a group of words that function together as a unit.

Types of Chunks:

• NP = Noun Phrase → “the big brown dog”
• VP = Verb Phrase → “is running fast”
• PP = Prepositional Phrase → “in the park”

Example:

“The big brown dog is running fast in the park.”

[The big brown dog] → NP (Noun Phrase)
[is running fast] → VP (Verb Phrase)
[in the park] → PP (Prepositional Phrase)

Chunking also uses BIO format, just like NER

The   → B-NP
big   → I-NP
brown → I-NP
dog   → I-NP
is    → B-VP
running → I-VP
fast  → I-VP
in    → B-PP
the   → I-PP
park  → I-PP

How is Chunking different from NER?

|                | NER                             | Chunking                          |
|----------------|---------------------------------|-----------------------------------|
| What it finds  | Named entities (people, places) | Grammatical phrases (NP, VP, PP)  |
| Focus          | Semantic (meaning)              | Syntactic (structure)             |
| Example        | "Paris" → LOCATION              | "The beautiful city" → NP         |

Use Cases of Chunking:

• 🔍 Information extraction — pulling subject-verb-object relationships
• 🤖 Question answering systems — identifying what a question is asking about
• 📝 Text summarization — understanding the key phrases in a document

Quick Code Demo:

import spacy

nlp  = spacy.load("en_core_web_sm")
text = "The big brown dog is running fast in the park."
doc  = nlp(text)

print("Noun Phrases (Chunks):")
for chunk in doc.noun_chunks:
    print(f"  '{chunk.text}' → {chunk.root.dep_}")

Output:

Noun Phrases (Chunks): 'The big brown dog' → nsubj 'the park' → pobj

# NER vs POS Tagging vs Chunking — The Big Comparison

![Comparison of NER, POS Tagging & Chunking](https://miro.medium.com/v2/resize:fit:1400/1*Gk0hUTS7Jq_1tP1g9EO_tA.png)

*Comparison of NER, POS Tagging & Chunking*

## One sentence summary:
- **NER asks:** “Who or what is this?”
- **POS asks: **“What type of word is this?”
- **Chunking asks:** “Which words belong together?”

— — —

# Bonus: How Does BERT Handle Token Classification?

Traditional NLP models processed text word by word from left to right. BERT (Bidirectional Encoder Representations from Transformers) changed everything by reading text in both directions at once.

## How BERT does token classification:

Input sentence → Tokenization → BERT Encoder → Linear Layer → Label per token

1. **Tokenization** : BERT uses WordPiece tokenizer which may split words into sub-words:
2. **BERT Encoder :** Each token gets a rich vector representation that captures context from the entire sentence
3. **Classification Head : **A simple linear layer maps each token’s vector to a label

# Final Thoughts

Token classification might sound like a dry, technical topic — but it’s the backbone of almost every intelligent text system we interact with daily.

Here’s what we covered today:

✅ **NLP** helps computers understand human language

✅ **Token Classification** labels each word in a sentence

✅ **NER** identifies real-world entities like people, places, and organizations

✅ **POS Tagging** reveals the grammatical role of each word

✅** Chunking** groups words into meaningful phrases

✅ **BERT **brings transformer power to all these tasks with incredible accuracy

Every time you use Google, chat with a bot, or get a smart autocomplete suggestion — token classification is quietly working behind the scenes. 🌟

— — —

Thanks for reading! If you found this helpful, give it a clap 👏 and follow for more beginner-friendly AI content.

— — —