🧠 Neural Storyteller: Building an Image Captioning Model Using Seq2Seq and Deep Learning

Introduction

What if machines could look at an image and describe it in natural human language?

This is exactly the problem solved by Image Captioning, a fascinating intersection of Computer Vision and Natural Language Processing (NLP). In this project, I built a multimodal deep learning system called Neural Storyteller that generates meaningful captions for images using a Sequence-to-Sequence (Seq2Seq) architecture.

This article walks through the architecture, training strategy, evaluation results, and lessons learned while building this system.

🚀 Project Objective

The goal of Neural Storyteller is simple yet challenging:

Convert visual information from an image into a grammatically correct natural language description.

To achieve this, the system must:

• Understand visual objects and context
• Translate visual features into language representations
• Generate fluent, human-like captions

📂 Dataset: Flickr30k

The model was trained using the Flickr30k dataset, which contains:

• ~31,000 real-world images
• Multiple human-written captions per image
• Diverse scenes including people, animals, sports, and everyday activities

Using multiple captions per image helps the model learn different ways of describing the same scene.

🏗️ Model Architecture

The system follows a multimodal encoder-decoder pipeline.

🔹 Step 1: Feature Extraction Using ResNet50

Training a Convolutional Neural Network alongside an RNN is computationally expensive. To solve this, I used a pre-trained ResNet50 model to extract visual features.

Instead of classifying images, the final classification layer was removed, allowing the network to output:

👉 A 2048-dimensional feature vector representing each image.

These features were cached to significantly speed up training.

🔹 Step 2: Seq2Seq Caption Generator

The caption generator consists of two components:

Encoder

The encoder transforms image features into a format suitable for the language model.

• Input: 2048-dimensional feature vector
• Output: Hidden state representation

A linear projection layer maps image features to the hidden dimension used by the decoder.

Decoder

The decoder is responsible for generating captions word-by-word using an LSTM network.

• Uses word embeddings as input
• Receives encoder output as initial hidden state
• Predicts next word until an end-of-sentence token is generated

🔍 Caption Generation Strategies

Two inference strategies were implemented:

Greedy Search

Selects the highest probability word at each step. Fast but may miss better sentence structures.

Beam Search

Maintains multiple candidate sequences simultaneously. Slower but generally produces higher-quality captions.

⚙️ Training Setup

• Loss Function: CrossEntropy Loss
• Optimizer: Adam
• Hardware: Kaggle GPU (T4 x2)
• Padding tokens ignored during training
• Word-level tokenization used

📊 Quantitative Evaluation

Evaluating image captioning is challenging because multiple correct captions may exist for a single image. Therefore, several NLP evaluation metrics were used.

BLEU Score

BLEU measures n-gram overlap between predicted captions and reference captions.

Corpus BLEU Scores

MetricScoreBLEU-10.1895BLEU-20.1046BLEU-30.0593BLEU-40.0336

Average Sentence BLEU-4

0.0405 ± 0.0562

BLEU-4 is the strictest metric since it measures 4-gram overlap. Lower scores are expected in captioning tasks due to language variability.

METEOR Score

METEOR improves upon BLEU by considering synonyms and word stems.

0.1724 ± 0.1298

This indicates the model is capturing semantic meaning beyond exact word matching.

Token-Level Metrics

These metrics evaluate vocabulary overlap between predicted and ground-truth captions.

MetricScorePrecision0.3978Recall0.2278F1 Score0.2764

📌 Evaluation Insights

Precision (39.78%)

Shows the model often predicts relevant words.

Recall (22.78%)

Indicates the model still misses some important descriptive words.

F1 Score (27.64%)

Highlights balanced but improvable caption quality.

Overall, the results demonstrate that the model successfully learns basic visual-to-text translation but still struggles with complex scene understanding.

🌐 Deployment

To make the model interactive, I built a Gradio Web Interface where users can:

• Upload an image
• Generate captions in real time
• Compare Greedy vs Beam Search outputs

This makes the project accessible beyond notebooks and research environments.

💡 Key Challenges Faced

1. Multimodal Representation Learning

Mapping visual features into language space is non-trivial.

2. Caption Diversity

Multiple valid captions exist for the same image, making evaluation difficult.

3. Training Stability

Balancing vocabulary size, embedding quality, and sequence length required experimentation.

🎯 Key Learnings

This project strengthened my understanding of:

• Multimodal deep learning
• Seq2Seq architectures
• CNN feature extraction
• NLP evaluation metrics
• AI model deployment

🔮 Future Improvements

Several enhancements could significantly improve performance:

• Adding attention mechanisms
• Using Transformer-based caption models
• Incorporating CIDEr and ROUGE evaluation metrics
• Training with larger datasets
• Real-time video captioning

Conclusion

Neural Storyteller demonstrates how deep learning can bridge the gap between vision and language. While the model still has room for improvement, it successfully generates meaningful captions and showcases the power of multimodal AI systems.

Image captioning remains an active research area, and advancements in transformers and attention mechanisms continue pushing the boundaries of what machines can describe.

If you’re working on multimodal AI, computer vision, or NLP, I would love to connect and discuss ideas!