The Bottleneck: Efficient Transfer Learning on CIFAR-10 using MobileNetV2

https://www.pickl.ai/blog/introduction-to-transfer-learning/

Abstract

This project demonstrates an optimized transfer learning pipeline to classify the CIFAR-10 dataset achieving $\ge$ 87% accuracy. By leveraging a pre-trained MobileNetV2 architecture and decoupling the feature extraction phase from the classification training phase, the model’s training time was drastically reduced. The final deliverable is a fully compiled, end-to-end Keras model that dynamically resizes inputs and maps them to 10 classes with high computational efficiency.

Introduction (What is the problem?)

Classifying 32x32 pixel images from the CIFAR-10 dataset presents a dual challenge: training a deep convolutional network from scratch is computationally expensive and prone to overfitting, while applying pre-trained Keras Applications natively designed for 224x224 ImageNet data requires architectural adjustments. Furthermore, naively passing datasets through frozen, deep architectures during every training epoch creates a severe computational bottleneck, wasting resources on static recalculations. The goal was to achieve high validation accuracy while designing a highly efficient training pipeline.

Materials and Methods (How did I solve the problem?)

All model development and training were executed first locally on a Linux environment (CachyOS) using TensorFlow/Keras, then on Google Collab with T4 GPU.

1. Data Preprocessing: Raw CIFAR-10 matrices were formatted using the native mobilenet_v2.preprocess_input function, and labels were one-hot encoded.
2. Architectural Adjustment: A Keras Lambda layer utilizing tf.image.resize was implemented as the true input layer to upscale the 32x32 images to the 224x224 dimensions expected by MobileNetV2.
3. The Bottleneck Optimization: To bypass the overhead of running MobileNetV2’s 150+ layers during every epoch, the base model was used purely for inference (.predict()) prior to training. This mapped the entire CIFAR-10 dataset into static feature vectors.
4. Top Classifier: A lightweight network consisting of GlobalAveragePooling2D, Dense(256) with ReLU, and a 30% Dropout layer was compiled.
5. Training: This top network was trained directly on the static feature vectors using the Adam optimizer, ReduceLROnPlateau, and EarlyStopping callbacks. Finally, the pre-trained base and the newly trained head were reassembled into a single computational graph.

Results (What did I find out?)

• Computational Speed: Extracting the bottleneck features required exactly one forward pass of the dataset. Consequently, training the top classifier took mere seconds per epoch, compared to the hours it would have taken to train end-to-end.
• Performance: The lightweight classifier successfully converged, mapping MobileNetV2’s generalized feature maps to CIFAR-10’s specific classes. The final reassembled model successfully loaded and evaluated at >90% validation accuracy in the automated test script.

Discussion

The success of this pipeline highlights a crucial principle in machine learning engineering: separating static computational graphs from dynamic ones. Because the weights of MobileNetV2 were frozen, its output for any given image was deterministic. By caching these outputs as feature vectors, we effectively treated MobileNetV2 as an advanced data preprocessing step rather than an active participant in the training loop. This project proves that high-performance hardware is not strictly necessary for complex deep learning tasks if the software architecture is optimized to prevent redundant tensor operations.