NLP Token Classification: Teaching Machines to Read Between the Lines

Imagine reading the sentence: “Hideo Kojima founded Kojima Productions in Tokyo after leaving Konami in 2015.”

As a human, you instantly know that Hideo Kojima is a person, Kojima Productions is an organization, and Tokyo is a location. You didn’t think twice about it.

But how do we teach a machine to do the same?

That’s exactly what Token Classification in Natural Language Processing (NLP) is all about — a foundational technique that lets machines label every single word (or “token”) in a sentence with a meaningful category.

In this blog, we’ll break down what token classification is, how it works under the hood, the labeling formats used, popular real-world applications, and how to get started with modern tools. No prior NLP expertise needed.

What Is Token Classification?

Token classification is a natural language understanding task where a label is assigned to each token (word or subword) in a piece of text. Think of it as putting a sticky note on every word in a sentence, describing what role that word plays.

What is a token? A token is the basic unit of text that a model processes. It can be a full word, part of a word, or even a single character — depending on the tokenizer used.

The two most popular token classification subtasks are:

• Named Entity Recognition (NER) — identifies real-world entities like people, places, organizations, and dates.
• Part-of-Speech (PoS) Tagging — labels words by their grammatical role: noun, verb, adjective, etc.

Both tasks follow the same principle: look at each token in context, and assign it the most appropriate label.

How Does Token Classification Work? (Step by Step)

Modern token classification pipelines follow three core stages.

Stage 1 — Tokenization

The raw sentence is broken into individual tokens. Traditional NLP models split on whitespace and punctuation. Modern transformer-based models like BERT use subword tokenization (WordPiece), which splits rare or complex words into smaller fragments.

For example, the name "Kojima" might be tokenized as ["Ko", "##jima"] if the model hasn't seen it frequently enough. This allows models to handle vocabulary they've never encountered before — even niche proper nouns from the world of game development.

Stage 2 — Contextual Encoding

Each token is converted into a dense numerical vector. Pre-trained language models like BERT are especially powerful here because they encode context. The word “Metal” gets a very different vector when surrounded by “Gear Solid” versus “steel alloy.”

Stage 3 — Per-Token Classification

A small classifier layer sits on top of the language model and predicts a label for each token independently. The model is fine-tuned on labeled datasets where every token already has the correct label.

The IOB Tagging Scheme

One of the most widely used labeling formats in NER is the IOB (Inside-Outside-Beginning) scheme, also called BIO tagging.

Instead of just saying “Kojima Productions is an organization,” IOB encoding tells the model exactly where a multi-word entity starts and ends:

• B-TYPE — Beginning of an entity of that type
• I-TYPE — Inside (continuation of) the same entity
• O — Outside — not part of any entity

Example: “Hideo Kojima directed Death Stranding.”

• Hideo → B-PER (beginning of a person name)
• Kojima → I-PER (inside the same person name)
• directed → O (not an entity)
• Death → B-WORK (beginning of a creative work title)
• Stranding → I-WORK (inside the same title)

This scheme helps models correctly identify multi-word entities like “Kojima Productions” or “Metal Gear Solid V: The Phantom Pain” as single units rather than treating each word separately.

Visual Example: NER in Action

Let’s take this sentence:

“Hideo Kojima, the creator of Metal Gear, launched Kojima Productions in Tokyo in December 2015.”

After running Named Entity Recognition, the output would look like this:

• Hideo Kojima → PERSON
• Metal Gear → WORK
• Kojima Productions → ORG
• Tokyo → LOC
• December 2015 → DATE

Everything else — “the creator of”, “launched”, “in” — gets labeled O (Outside). The model has successfully pulled structured facts from an otherwise unstructured sentence.

Code Example: NER with Hugging Face Transformers

For state-of-the-art accuracy using a BERT-based model:

from transformers import pipeline
ner = pipeline("ner", model="dslim/bert-base-NER",
               aggregation_strategy="simple")

results = ner("Hideo Kojima won the BAFTA Games Award for Death Stranding in London.")

for r in results:
    print(r["word"], "→", r["entity_group"], f"(confidence: {r['score']:.2f})")

Output:

Hideo Kojima     →  PER   (confidence: 0.99)
BAFTA            →  ORG   (confidence: 0.97)
Death Stranding  →  MISC  (confidence: 0.95)
London           →  LOC   (confidence: 0.98)

The Hugging Face pipeline handles tokenization, model inference, and label alignment — making it beginner-friendly while using production-grade models under the hood.

Real-World Use Case: Building a Kojima Knowledge Graph

Imagine you’re building a fan knowledge base about Hideo Kojima’s career — pulling data from interviews, Wikipedia articles, and game reviews. Manually tagging every mention of a game title, studio, award, or collaborator would take weeks.

With token classification, you can automate it entirely.

Feed in a sentence like “Norman Reedus stars in Death Stranding, developed by Kojima Productions and published by Sony Interactive Entertainment.” and NER automatically extracts:

• Norman Reedus → PERSON (actor/collaborator)
• Death Stranding → WORK (game title)
• Kojima Productions → ORG (developer)
• Sony Interactive Entertainment → ORG (publisher)

Do this across thousands of documents and you’ve built a rich, structured knowledge graph that can power search, recommendations, and Q&A — all without a single human manually tagging a word.

This same pattern applies across industries: healthcare extracts drug names and symptoms from clinical notes, legal tech identifies parties and dates in contracts, and finance platforms track executive names and stock tickers in earnings call transcripts.

Common Challenges

Token classification is powerful, but not without its hurdles.

Ambiguity is the trickiest. Consider the word “Kojima” — in most gaming contexts it refers to Hideo Kojima the person, but “Kojima Productions” is an organization. Without sufficient surrounding context, a model can easily confuse the two. This is label ambiguity, and it’s one of the hardest problems in NER.

Imbalanced labels are another issue. In most real-world text, the vast majority of tokens are ordinary words labeled O. This makes models biased toward predicting "not an entity" for everything. Oversampling techniques and weighted loss functions help address this.

Domain mismatch is a frequent pain point. A general-purpose model trained on news articles will struggle to recognize “CODEC” as a communication device in a Metal Gear context, or “VAMP” as a character name rather than a common noun. Fine-tuning on domain-specific data is the standard solution.

Subword alignment causes subtle bugs. When a tokenizer splits “Stranding” into multiple subword pieces, the model predicts a label for each piece. Mapping those predictions back to the original word requires careful alignment logic — a common source of errors in NER pipelines.

Popular Tools to Explore

spaCy is the best starting point for most beginners — fast, well-documented, and production-ready with pre-trained models for English and many other languages.

Hugging Face Transformers is the go-to library for fine-tuning BERT, RoBERTa, or other transformer models on custom NER datasets. Their pipeline API makes inference extremely accessible.

Flair is worth exploring if you want state-of-the-art accuracy — it uses contextual string embeddings and consistently ranks at the top of NER benchmarks.

NLTK is great for learning the classical foundations of NER before jumping into deep learning approaches.

For specialized domains, BioBERT handles biomedical text and LegalBERT excels on legal documents — both are fine-tuned variants of BERT trained on domain-specific corpora.

Conclusion

Token classification is one of the most practical and widely deployed building blocks in modern NLP. Whether you’re extracting named entities from game reviews, tagging parts of speech for a grammar tool, or helping a knowledge base understand that “Kojima” refers to a person in one sentence and a studio in another — this technique is working quietly behind the scenes.

The core idea is elegant: take each word, understand it in context, and assign it a label. But the execution — transformer-based contextual encoding, IOB tagging schemes, and domain-specific fine-tuning — is what makes it genuinely powerful.

Where to go from here:

1. Run pip install spacy && python -m spacy download en_core_web_sm and try NER on a paragraph about your favourite game.
2. Browse the Hugging Face token classification models.
3. Try fine-tuning BERT on a custom NER dataset using the Hugging Face Trainer API.

That curiosity — that first sentence you feed into a model and see labeled back — is the beginning of your NLP journey. Even Kojima started somewhere.