Transfer Learning in Computer Vision: From Pretrained Models to Production-Ready Systems

Photo by Chris Liverani on Unsplash

Why This Blog Exists (And Why Transfer Learning Is Not Optional)

Training deep learning models from scratch sounds impressive.

In production, it’s usually a mistake.

Modern computer vision systems are built on pretrained backbones — models trained on massive datasets like ImageNet that already understand how the visual world works. The real engineering challenge is adapting that knowledge efficiently and correctly to new tasks.

This blog walks through a complete, hands-on transfer learning workflow using TorchVision — starting from a pretrained model that completely fails on handwritten digits and systematically transforming it into a high-accuracy classifier.

This is not theory.

This is how real CV systems are built.

The Core Problem: When Pretrained Models Fail Spectacularly

Let’s start with a simple experiment.

A MobileNetV3 model trained on ImageNet is exceptionally good at identifying real-world objects. But what happens when we show it handwritten digits from the EMNIST dataset?

It predicts:

• “spatula”
• “bolo tie”
• “nematode”

Confidently.

This failure is not surprising. The model has never learned what a digit is. It simply maps unfamiliar shapes to the closest visual patterns it knows.

This exposes the fundamental limitation of pretrained models:

They provide visual understanding, not task understanding.

Transfer learning is how we fix that.

Transfer Learning: The Mental Model

Every pretrained vision model can be split into two conceptual parts:

1. Feature Extractor (Backbone)

• Early and mid-level convolutional layers
• Learns edges, curves, textures, shapes
• Highly reusable across domains

2. Classifier Head

• Final layers
• Maps features to task-specific labels
• Must be replaced for new tasks

Transfer learning means:

1. Replace the classifier head
2. Decide how much of the backbone to retrain

That decision defines the strategy.

Tooling and Setup

import torch
import torch.nn as nn
import torchvision.models as tv_models
import torchvision.transforms as transforms

import helper_utils

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

Step 1: Preparing the Data (This Actually Matters)

Pretrained models expect very specific input formats. If you violate them, performance collapses silently.

EMNIST → ImageNet-Compatible Pipeline

emnist_transformation = transforms.Compose([
    transforms.Grayscale(num_output_channels=3),
    transforms.Resize((224, 224)),
    transforms.RandomRotation(degrees=(90, 90)),
    transforms.RandomVerticalFlip(p=1.0),
    transforms.ToTensor(),
    transforms.Normalize(
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
])

Key decisions:

• Convert grayscale → RGB
• Match ImageNet resolution
• Normalize with ImageNet statistics
• Apply domain-appropriate augmentation

Step 2: Proving the Failure Case

mobilenet_model = tv_models.mobilenet_v3_small(
    weights="IMAGENET1K_V1"
).eval()

class_names = helper_utils.load_imagenet_classes(
    "./imagenet_class_index.json"
)

helper_utils.show_predictions(
    mobilenet_model,
    val_loader,
    device,
    class_names
)

Result: total semantic mismatch.

This is the baseline failure that motivates transfer learning.

Transfer Learning Strategy 1: Feature Extraction

What We Do

• Freeze the entire backbone
• Replace the classifier head
• Train only the new head

Why It Works

The backbone already extracts powerful visual features. The new head simply learns how to interpret them for a new task.

Architecture Pattern 1: Direct Attribute (ResNet)

resnet18_model = tv_models.resnet18(weights="IMAGENET1K_V1")

# Freeze entire backbone
for param in resnet18_model.parameters():
    param.requires_grad = False

# Replace classifier
num_features = resnet18_model.fc.in_features
resnet18_model.fc = nn.Linear(num_features, 5)

ResNet exposes its classifier directly as model.fc.

This is the simplest case.

Architecture Pattern 2: Modular Block (MobileNetV3)

mobilenet_model = tv_models.mobilenet_v3_small(
    weights="IMAGENET1K_V1"
)

# Freeze backbone
for param in mobilenet_model.features.parameters():
    param.requires_grad = False

# Replace classifier head
num_features = mobilenet_model.classifier[-1].in_features
mobilenet_model.classifier[-1] = nn.Linear(num_features, 10)

Here, the backbone lives inside features, and the classifier is a Sequential block.

Understanding this distinction is critical.

Training Only the New Head

loss_function = nn.CrossEntropyLoss()

optimizer = torch.optim.SGD(
    filter(lambda p: p.requires_grad, mobilenet_model.parameters()),
    lr=0.001
)

trained_model = helper_utils.training_loop(
    model=mobilenet_model,
    trainloader=train_loader,
    valloader=val_loader,
    loss_function=loss_function,
    optimizer=optimizer,
    num_epochs=1,
    device=device
)

Result:

Over 80% validation accuracy in a single epoch.

This is why transfer learning dominates real-world CV.

Transfer Learning Strategy 2: Fine-Tuning

Feature extraction assumes pretrained features are sufficient. Often, they are not.

Fine-tuning allows partial adaptation.

The Two-Stage Best Practice

1. Train classifier head only
2. Unfreeze top backbone layers and continue training with a lower learning rate

Unfreezing the Top Layers (MobileNetV3)

fine_tune_model = trained_model

# Unfreeze last feature block
for param in fine_tune_model.features[12].parameters():
    param.requires_grad = True

Only high-level features adapt — low-level features remain stable.

Fine-Tuning Optimizer

optimizer = torch.optim.SGD(
    filter(lambda p: p.requires_grad, fine_tune_model.parameters()),
    lr=1e-5
)

fine_tuned_model = helper_utils.training_loop(
    model=fine_tune_model,
    trainloader=train_loader,
    valloader=val_loader,
    loss_function=loss_function,
    optimizer=optimizer,
    num_epochs=1,
    device=device
)

Accuracy improves further, at the cost of longer training time.

This is the most common production strategy.

Transfer Learning Strategy 3: Full Retraining

When you have:

• Large datasets
• Significant domain shift
• High performance requirements

You retrain everything.

Unfreeze the Entire Model

full_retrain_model = fine_tuned_model

for param in full_retrain_model.parameters():
    param.requires_grad = True

Full Training

optimizer = torch.optim.SGD(
    full_retrain_model.parameters(),
    lr=1e-4
)

final_model = helper_utils.training_loop(
    model=full_retrain_model,
    trainloader=train_loader,
    valloader=val_loader,
    loss_function=loss_function,
    optimizer=optimizer,
    num_epochs=1,
    device=device
)

Highest flexibility. Highest cost.

What This Project Actually Demonstrates

Beyond digits and accuracy, this work shows:

• Architectural literacy across TorchVision models
• Correct freezing and optimizer scoping
• Controlled fine-tuning strategies
• Data preprocessing discipline
• Production-grade decision making

These are engineering skills, not course exercises.

Real-World Applications

This exact workflow applies to:

• Medical imaging
• OCR systems
• Defect detection
• Retail vision
• Satellite imagery
• Security systems

Change the dataset.

Change the head.

Choose the strategy.

Final Thoughts

Transfer learning is not a shortcut.

It is how serious computer vision systems are built.

Knowing when to freeze, when to fine-tune, and when to retrain fully is a skill that separates practitioners from experimenters.