Building an End-to-End NLP Pipeline (From Text to Deployment)

So far in this series, we’ve learned about:

• Text preprocessing
• Vectorization
• Transformers (BERT, GPT)
• Training and inference

Now it’s time to connect all the pieces.

In this article, we’ll walk through a complete NLP pipeline — from raw text to a deployed model.

What Is an NLP Pipeline?

An NLP pipeline is the sequence of steps that turns raw text into useful predictions.

High-level view:

Raw Text
 → Cleaning
 → Tokenization
 → Vectorization
 → Model
 → Evaluation
 → Inference / Deployment

Every real-world NLP system follows this structure.

1. Data Collection & Cleaning

Data Sources

Common text data sources:

• CSV files (reviews, comments)
• Databases
• APIs (Twitter, Reddit)
• Logs, documents, PDFs

Example dataset (sentiment analysis):

| Text                  | Label    |
| --------------------- | -------- |
| "I love this product" | Positive |
| "Terrible experience" | Negative |

Basic Text Cleaning

Goals:

• Remove noise
• Normalize text
• Keep meaning intact

import re

def clean_text(text):
    text = text.lower()
    text = re.sub(r"http\S+", "", text)   # remove URLs
    text = re.sub(r"[^a-z\s]", "", text)  # remove punctuation
    text = re.sub(r"\s+", " ", text).strip()
    return text

clean_text("I LOVE NLP!!! Visit https://example.com")

✔ Keep cleaning minimal for transformer models ✔ Avoid over-cleaning (important for meaning)

2. Tokenization Strategies

Tokenization converts text into tokens the model understands.

Word-Level Tokenization (Traditional)

Used in:

• Bag of Words
• TF-IDF

text = "I love NLP"
tokens = text.split()
print(tokens)

Limitation:

• Vocabulary explosion
• Poor handling of unknown words

Subword Tokenization (Modern — Recommended)

Used by:

• BERT
• GPT
• T5

Advantages:

• Handles unknown words
• Smaller vocabulary
• Better generalization

from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
tokens = tokenizer.tokenize("Unbelievable performance")
print(tokens)

Tokenization Choice Summary

| Model Type       | Tokenization |
| ---------------- | ------------ |
| BoW / TF-IDF     | Word-level   |
| BERT / GPT       | Subword      |
| Character models | Char-level   |

3. Model Selection

Choose your model based on the task, not hype.

Common NLP Tasks → Models

| Task                | Recommended Model |
| ------------------- | ----------------- |
| Text classification | BERT              |
| NER                 | BERT              |
| Semantic search     | Sentence-BERT     |
| Text generation     | GPT               |
| Summarization       | T5 / BART         |
| Translation         | Encoder–Decoder   |

Example: Choosing BERT for Classification

Why?

• Strong contextual understanding
• Pretrained knowledge
• Easy fine-tuning

4. Training the Model

We’ll show a simple fine-tuning example using BERT for sentiment classification.

Step 1: Load Model & Tokenizer

from transformers import BertTokenizer, BertForSequenceClassification

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=2
)

Step 2: Prepare Data

texts = ["I love this movie", "This was terrible"]
labels = [1, 0]

inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    return_tensors="pt"
)

Step 3: Training Loop (Simplified)

import torch

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)

outputs = model(**inputs, labels=torch.tensor(labels))
loss = outputs.loss

loss.backward()
optimizer.step()

✔ In practice, use Trainer or PyTorch Lightning ✔ This shows the core idea

5. Evaluation

Evaluation ensures your model actually works.

| Task           | Metrics           |
| -------------- | ----------------- |
| Classification | Accuracy, F1      |
| NER            | Precision, Recall |
| Generation     | BLEU, ROUGE       |
| Search         | Cosine similarity |

Simple Accuracy Example

preds = outputs.logits.argmax(dim=1)
accuracy = (preds == torch.tensor(labels)).float().mean()
print("Accuracy:", accuracy.item())

6. Inference (Using the Model)

Inference is what happens after training, when users interact with your model.

text = "I really enjoyed this product"

inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)

prediction = outputs.logits.argmax(dim=1).item()
print("Prediction:", "Positive" if prediction == 1 else "Negative")

7. Deployment Overview

Deployment means making your model usable.

| Option             | Use Case          |
| ------------------ | ----------------- |
| REST API (FastAPI) | Web & mobile apps |
| Batch jobs         | Offline analysis  |
| Cloud services     | Scalability       |
| Edge devices       | On-device NLP     |

Simple FastAPI Sketch

from fastapi import FastAPI

app = FastAPI()

@app.post("/predict")
def predict(text: str):
    inputs = tokenizer(text, return_tensors="pt")
    outputs = model(**inputs)
    label = outputs.logits.argmax(dim=1).item()
    return {"label": label}

8. Monitoring & Iteration (Often Ignored)

Real-world pipelines must handle:

• Data drift
• Performance drops
• New language patterns

Best practices:

• Log predictions
• Monitor metrics
• Retrain periodically

Complete NLP Pipeline Summary

Collect data
 → Clean text
 → Tokenize
 → Vectorize
 → Train model
 → Evaluate
 → Deploy
 → Monitor & improve

Every production NLP system follows this loop.

Key Takeaways

• NLP pipelines connect theory to real-world systems
• Tokenization and model choice are critical
• Transformers simplify feature engineering
• Evaluation and deployment matter as much as training
• NLP is an iterative process, not a one-time task

Evaluation Metrics for NLP Tasks (With Intuition and Examples)

Training an NLP model is only half the job. The real question is:

How do we know if our model is actually good?

That’s where evaluation metrics come in.

In this article, we’ll break down the most important NLP metrics, explain when to use them, and show why choosing the wrong metric can be misleading.

Why Evaluation Metrics Matter

Two models can have the same accuracy but behave very differently in real life.

Example:

• Spam detection
• Medical text analysis
• Toxic content moderation

In these cases, some mistakes matter more than others.

Accuracy — The Simplest Metric

What Is Accuracy?

Accuracy measures:

Correct predictions / Total predictions

Simple Example

correct = 90
total = 100
accuracy = correct / total
print(accuracy)

Output:

0.9

✔ Easy to understand ❌ Often misleading

Why Accuracy Can Be Misleading

Imagine a dataset:

ClassCountNot Spam95Spam5

If a model predicts “Not Spam” every time:

Accuracy = 95%

But:

• It catches zero spam
• Completely useless

➡ Accuracy ignores class imbalance.

Precision and Recall (Core Concepts)

To fix accuracy’s weaknesses, we use precision and recall.

Precision — “How Careful Am I?”

Of all predicted positives, how many were correct?

Precision = TP / (TP + FP)

Example:

• Model flags 10 emails as spam
• Only 6 are actually spam

Precision = 6 / 10 = 0.6

✔ High precision → fewer false alarms

Recall — “How Much Did I Catch?”

Of all actual positives, how many did I find?

Recall = TP / (TP + FN)

Example:

• There are 20 spam emails
• Model finds 6

Recall = 6 / 20 = 0.3

✔ High recall → fewer misses

Precision vs Recall Intuition

| Scenario          | Focus     |
| ----------------- | --------- |
| Spam filtering    | Precision |
| Medical diagnosis | Recall    |
| Toxic content     | Recall    |
| Search results    | Precision |

F1-Score — The Balance

F1-score combines precision and recall:

F1 = 2 × (Precision × Recall) / (Precision + Recall)

Why F1 Is Important

• Penalizes extreme imbalance
• Useful for uneven datasets
• Common in NLP tasks

F1 Code Example

from sklearn.metrics import f1_score

y_true = [1, 0, 1, 1, 0]
y_pred = [1, 0, 0, 1, 0]

print(f1_score(y_true, y_pred))

Accuracy vs F1-Score (Quick Comparison)

| Metric    | Best For              |
| --------- | --------------------- |
| Accuracy  | Balanced datasets     |
| F1-score  | Imbalanced datasets   |
| Precision | Avoid false positives |
| Recall    | Avoid false negatives |

BLEU and ROUGE Explained in Detail

When NLP models generate text (translations, summaries, captions), evaluation becomes tricky.

Unlike classification:

• There is no single correct answer
• Many outputs can be equally valid

BLEU and ROUGE are two of the most widely used automatic metrics for this purpose.

Why We Need BLEU and ROUGE

Generated text must be compared against human-written reference text.

Example:

Reference: "The cat is sitting on the mat"
Candidate: "A cat sits on the mat"

Meaning is similar, wording is different.

BLEU and ROUGE measure overlap, not true understanding — but they’re still useful.

BLEU (Bilingual Evaluation Understudy)

What Is BLEU?

BLEU is primarily used for:

• Machine translation
• Text generation

BLEU measures:

How much of the generated text appears in the reference text

BLEU is precision-focused.

Core Idea of BLEU

“Are the words (and phrases) I generated present in the reference?”

BLEU checks n-gram overlap:

• Unigrams (1 word)
• Bigrams (2 words)
• Trigrams (3 words)
• 4-grams

BLEU Step-by-Step Example

Reference Sentence

"the cat is on the mat"

Generated Sentence

"the cat sat on mat"

Step 1: Unigram Precision (BLEU-1)

Generated unigrams:

the, cat, sat, on, mat

Reference unigrams:

the, cat, is, on, the, mat

Matching words:

the, cat, on, mat → 4 matches

BLEU-1:

4 / 5 = 0.8

Step 2: Bigram Precision (BLEU-2)

Generated bigrams:

the cat
cat sat
sat on
on mat

Reference bigrams:

the cat
cat is
is on
on the
the mat

Matching bigrams:

the cat → 1 match

BLEU-2:

1 / 4 = 0.25

Step 3: Combine N-grams

BLEU combines:

• BLEU-1
• BLEU-2
• BLEU-3
• BLEU-4

Using geometric mean:

BLEU = exp( average(log(n-gram precisions)) )

Higher-order n-grams enforce fluency and word order.

Brevity Penalty (Important!)

BLEU penalizes overly short outputs.

Example:

Reference: "the cat is on the mat"
Candidate: "cat mat"

Unigram precision = high But meaning = poor

BLEU applies brevity penalty to reduce score.

BLEU Code Example

from nltk.translate.bleu_score import sentence_bleu

reference = [["the", "cat", "is", "on", "the", "mat"]]
candidate = ["the", "cat", "sat", "on", "mat"]

score = sentence_bleu(reference, candidate)
print(score)

BLEU Strengths

✔ Easy to compute ✔ Correlates reasonably with human translation quality ✔ Widely accepted benchmark

BLEU Limitations

❌ Precision-heavy (ignores recall) ❌ Penalizes paraphrases ❌ Does not measure meaning ❌ Sensitive to exact wording

ROUGE (Recall-Oriented Understudy for Gisting Evaluation)

What Is ROUGE?

ROUGE is mainly used for:

• Text summarization
• Content coverage tasks

ROUGE is recall-focused.

Core Idea of ROUGE

“How much of the reference text did I capture?”

ROUGE checks how much overlap exists between generated and reference text.

Common ROUGE Variants

| Variant | Measures                   |
| ------- | -------------------------- |
| ROUGE-1 | Unigram recall             |
| ROUGE-2 | Bigram recall              |
| ROUGE-L | Longest common subsequence |

ROUGE-1 Example

Reference Summary

"the cat is on the mat"

Generated Summary

"the cat on mat"

Matching unigrams:

the, cat, on, mat → 4

ROUGE-1 Recall:

4 / 6 = 0.67

ROUGE-2 Example

Reference bigrams:

the cat
cat is
is on
on the
the mat

Generated bigrams:

the cat
cat on
on mat

Matching bigrams:

the cat → 1

ROUGE-2:

1 / 5 = 0.2

ROUGE-L (Longest Common Subsequence)

ROUGE-L measures:

• Longest word sequence appearing in both texts
• Order matters
• Gaps allowed

Example:

Reference: the cat is on the mat
Generated: the cat on mat

LCS:

the → cat → on → mat

Length = 4

ROUGE Code Example

from rouge_score import rouge_scorer

scorer = rouge_scorer.RougeScorer(["rouge1", "rouge2", "rougeL"], use_stemmer=True)

reference = "the cat is on the mat"
candidate = "the cat on mat"

scores = scorer.score(reference, candidate)
print(scores)

ROUGE Strengths

✔ Measures content coverage ✔ Works well for summarization ✔ Recall-focused

ROUGE Limitations

❌ Ignores paraphrasing ❌ Over-rewards long outputs ❌ Does not judge coherence or factuality

When to Use What

| Task                  | Metric                   |
| --------------------- | ------------------------ |
| Machine translation   | BLEU                     |
| Summarization         | ROUGE                    |
| Paraphrasing          | Neither alone            |
| Open-ended generation | Human + semantic metrics |

Key Takeaways

• BLEU measures how precise generated text is
• ROUGE measures how complete generated text is
• Both rely on n-gram overlap
• Neither understands meaning
• Use them wisely, not blindly