Deep Learning Architecture 1 : Lenet

LeNet Architecture: A Foundation of Convolutional Neural Networks

The LeNet architecture, created by Yann LeCun and his team in the late 1980s, was a groundbreaking model for image recognition tasks. Its inception laid the foundation for modern convolutional neural networks (CNNs) by introducing techniques like convolutional layers, pooling layers, and fully connected layers in a hierarchical design that captures spatial hierarchies in images. LeNet has played a pivotal role in digit recognition and inspired numerous advancements in deep learning.

Key Details of LeNet

• Original Application: LeNet was developed to recognize handwritten digits, specifically for postal code digit recognition on checks.
• Dataset Used: It was first applied to the MNIST dataset, which consists of handwritten digits (0–9) and is widely used as a benchmark for image classification tasks.
• Competition: Although not initially developed for a competition, LeNet set a new benchmark for digit classification tasks and inspired early competitions in computer vision.

Architecture Overview

LeNet-5, the most famous version, consists of seven layers, including convolutional layers, pooling layers, and fully connected layers. Here’s a breakdown of each layer:

1. Input Layer: Accepts a grayscale image of size 32×32.
2. C1 — Convolutional Layer: Uses 6 filters of size 5×5 with a stride of 1, producing a feature map of size 28×28.
3. S2 — Subsampling (Pooling) Layer: Applies average pooling with a kernel size of 2×2 and a stride of 2, reducing the feature map to 14×14.
4. C3 — Convolutional Layer: Uses 16 filters of size 5×5 with a stride of 1, giving output feature maps of size 10×10.
5. S4 — Subsampling (Pooling) Layer: Again applies average pooling, resulting in feature maps of size 5×5.
6. C5 — Fully Connected Convolutional Layer: This layer has 120 neurons, each connected to all 5x5 feature maps from the previous layer.
7. F6 — Fully Connected Layer: Contains 84 neurons.
8. Output Layer: Uses a softmax activation function to output probabilities for 10 classes (digits 0–9).

Formula to calculate size of the image after performing convolution

Advantages of LeNet

• Efficient Feature Extraction: Convolutional layers efficiently capture spatial hierarchies in images, which makes the model particularly effective for image data.
• Reduced Parameter Count: Pooling layers reduce the spatial size, lowering the parameter count and computational load.
• Foundational Design: LeNet’s design introduced several principles that modern CNN architectures have built upon, including AlexNet, VGG, and ResNet.

Code:

import tensorflow as tf
from tensorflow.keras import layers, models, datasets
import numpy as np

# Load and preprocess the MNIST dataset
(x_train, y_train), (x_test, y_test) = datasets.mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0  # Normalize pixel values
x_train = np.expand_dims(x_train, -1)  # Add a channel dimension for grayscale
x_test = np.expand_dims(x_test, -1)

model = Sequential()
model.add(Conv2D(6, kernel_size=(5, 5), activation='tanh', input_shape=(28, 28, 1)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Conv2D(16, kernel_size=(5, 5), activation='tanh'))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Flatten())
model.add(Dense(120, activation='tanh'))
model.add(Dense(84, activation='tanh'))
model.add(Dense(10, activation='softmax'))

# Compile the model
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Train the model
model.fit(x_train, y_train, epochs=10, batch_size=64, validation_data=(x_test, y_test))

# Evaluate the model
test_loss, test_acc = model.evaluate(x_test, y_test)
print(f'Test Accuracy: {test_acc:.4f}')

Summary:

In the LeNet model, convolution, average pooling, and fully connected layers were introduced to the world.