Neural Storyteller: Teaching a Machine to Describe Images in Plain English

How We Built an Image Captioning System from Scratch Using ResNet-50, LSTM, and Seq2Seq Architecture on the Flickr30k Dataset

A deep-dive into multimodal deep learning — where Computer Vision meets Natural Language Processing.

The Big Idea

Imagine showing a photograph to someone who has never seen the world before — only numbers. Can they describe what they see?

That is exactly the challenge we tackled in this project. We built Neural Storyteller, a multimodal deep learning system that takes any image as input and generates a natural English description — completely automatically.

Feed it a photo of a golden retriever splashing through a river and it writes:

“a dog is running through the water”

Feed it a crowded street scene and it writes:

“a group of people are walking down a busy street”

This is not magic. It is an Encoder-Decoder architecture — a ResNet-50 CNN reading the image, and an LSTM network writing the words. Let us walk through exactly how we built it.

The Dataset: Flickr30k

We trained on the Flickr30k dataset — 31,000 real-world photographs, each annotated with 5 different human-written captions. That gives us approximately 155,000 caption-image pairs to learn from.

Why 5 captions per image? Because language is inherently subjective. One person sees “a man in a blue jacket”, another sees “a person standing near a wall”. Both are correct. Having multiple references makes training and evaluation more robust.

The dataset covers everything — sports, animals, people, nature, street scenes. This diversity is essential for a model that generalizes to real-world images.

The Architecture: A Tale of Two Networks

Our system follows the classic Show and Tell paradigm, introduced by Google in 2014. The core insight is elegant:

Treat caption generation as a translation problem — translate an image into a sentence.

This gives us two components working in tandem:

1. The Encoder — Making Sense of Pixels

We used a pre-trained ResNet-50 as our visual encoder. ResNet-50 is a 50-layer deep convolutional neural network trained on ImageNet — 1.2 million images across 1000 categories. It already knows how to recognize edges, textures, shapes, animals, and objects.

The key trick: we remove the final classification layer. Instead of getting “golden retriever (92% confidence)”, we get a 2048-dimensional feature vector — a rich numerical fingerprint of the image that captures its visual content without committing to a single label.

Raw Image (224×224×3)
        ↓
    ResNet-50
        ↓
  2048-dim vector
  [0.23, -1.4, 0.87, ... 2048 numbers]

Since running ResNet-50 on 31,000 images is computationally expensive, we pre-compute and cache all features in a .pkl file. This means the CNN runs only once — during training, we just load the cached vectors. Smart engineering that saves hours of compute.

A thin Linear projection layer then compresses 2048 → 512 dimensions, matching the LSTM’s hidden size. Batch Normalization and ReLU are applied to stabilize the representation.

2. The Decoder — Writing the Words

The decoder is an LSTM (Long Short-Term Memory) network — a type of Recurrent Neural Network designed specifically for sequential data.

The LSTM generates captions word by word. At each step:

• It receives the previously generated word (as a learned embedding vector)
• It maintains a hidden state (short-term memory) and cell state (long-term memory)
• It outputs a probability distribution over the entire vocabulary
• The highest-probability word becomes the next output

The encoder’s output initializes the LSTM’s hidden state — this is how the image information “seeds” the text generation process.

Encoder Output (512-dim)
        ↓
   LSTM Hidden State Init
        ↓
<start> → LSTM → "a"
    ↓
   "a" → LSTM → "dog"
    ↓
  "dog" → LSTM → "runs"
    ↓
 "runs" → LSTM → <end>

Final Caption: "a dog runs"

Text Preprocessing: From Raw Captions to Numbers

Neural networks speak numbers, not English. We built a complete text processing pipeline:

Step 1 — Cleaning: Lowercase everything, remove punctuation and digits, collapse extra spaces. “A dog running!! near water 2024” becomes “a dog running near water”.

Step 2 — Tokenization: Split into word lists. “a dog running” → [“a”, “dog”, “running”].

Step 3 — Vocabulary Building: Count every word across all 155,000 captions. Keep only words appearing 5 or more times — rare words become <unk> (unknown). This gives us a vocabulary of ~8,000 words.

Step 4 — Special Tokens: Four special tokens anchor every encoded sequence:

• <pad> — fills sequences to a fixed length (index 0)
• <start> — tells the decoder "begin generating"
• <end> — tells the decoder "stop here"
• <unk> — represents any out-of-vocabulary word

Step 5 — Encoding: Every caption becomes a fixed-length sequence of 40 integers, padded or truncated as needed.

Step 6 — Data Split: We split by image (not by row) into 80% train, 10% validation, 10% test. Splitting by image is critical — otherwise the same image’s 5 captions could appear in both train and test, creating data leakage and falsely inflated scores.

Training: Teacher Forcing and CrossEntropy Loss

Training a caption model naively is hard. Early in training, the model makes mistakes. If we feed those wrong predictions back as the next input, errors cascade — the model never learns anything meaningful.

The solution is Teacher Forcing: during training, we always feed the ground truth previous word as input, regardless of what the model predicted. This stabilizes training dramatically.

For the caption [“a”, “dog”, “runs”, “fast”]:

• Input to decoder: [<start>, a, dog, runs]
• Expected output: [a, dog, runs, <end>]

At every position, the model predicts the next word. We use CrossEntropy Loss to measure how wrong each prediction was — with ignore_index=PAD_IDX so padding positions don't contribute to the loss.

Adam optimizer with lr=3e-4 drives the weight updates. Gradient clipping (max_norm=5.0) prevents exploding gradients during backpropagation through time — essential for LSTM stability. ReduceLROnPlateau halves the learning rate when validation loss plateaus, helping the model fine-tune in later epochs.

We trained for 25 epochs, saving the best model checkpoint based on validation loss.

Inference: How the Model Generates Captions

At inference time, there is no ground truth to feed — the model must rely on its own predictions. We implemented two decoding strategies:

Greedy Search

The simplest approach: at each step, pick the word with the highest probability. Fast and deterministic, but locally optimal — one bad early choice can derail the entire caption.

Beam Search

The smarter approach: instead of committing to one word, keep the top-5 candidate sequences alive at each step. Each beam is scored by the sum of log probabilities of all its chosen words (log probabilities are used instead of raw probabilities to avoid numerical underflow). Sequences are length-normalized to prevent bias toward shorter captions.

Beam search consistently produces more fluent and accurate captions than greedy search — at the cost of slightly more computation.

Results and Evaluation

We evaluated our model on the held-out test set using five metrics:

Metric Score What It Measures BLEU-4 ~0.22 4-gram overlap precision Token Precision ~0.58 Predicted words in reference Token Recall ~0.54 Reference words found in prediction Token F1 ~0.55 Harmonic mean of P and R METEOR ~0.38 Synonym-aware alignment ROUGE-L ~0.48 Longest common subsequence

BLEU-4 of ~0.22 is competitive for a vanilla Seq2Seq model without attention — state-of-the-art transformer-based models score 0.35+ but require far more compute and data.

Why so many metrics? Each captures something different. BLEU ignores synonyms — “happy” and “joyful” score 0 overlap despite identical meaning. METEOR uses WordNet to handle synonyms and stems. ROUGE-L checks if the words appear in the same order using Longest Common Subsequence. Together, they give a complete picture of caption quality.

Key Takeaways and Lessons Learned

Transfer Learning is powerful. ResNet-50 was never trained for captioning — yet its ImageNet features transfer beautifully to our task. We got a strong visual encoder for free.

Caching features saves massive compute. Pre-computing and saving the 2048-dim vectors means the CNN runs once instead of once per training batch per epoch. Hours of GPU time saved.

Data leakage is subtle but deadly. Splitting at the image level (not caption row level) is not optional — it is the difference between honest evaluation and misleading results.

Teacher forcing is a double-edged sword. It makes training fast and stable, but creates a gap with inference behavior (exposure bias). Scheduled sampling is a technique to bridge this gap — something to explore next.

Beam search beats greedy, always. The small compute overhead of maintaining 5 beams consistently delivers more coherent, complete captions.

What’s Next?

This Seq2Seq baseline is just the beginning. Modern image captioning has moved far beyond this:

• Attention Mechanisms — Let the model focus on different image regions when generating each word. “dog” → look at the dog region. “water” → look at the river region.
• Transformer-based models — Replace the LSTM with a Transformer decoder for better long-range dependencies.
• Vision-Language Models — CLIP, BLIP, and Flamingo use contrastive pre-training at massive scale to achieve much stronger multimodal understanding.
• Reinforcement Learning from Human Feedback — Fine-tune captions based on what humans actually prefer rather than n-gram overlap metrics.

Try It Yourself

We deployed the model as an interactive Gradio web app — upload any image and get an instant caption. The full source code, training pipeline, and evaluation notebooks are available on our GitHub repository.

“The goal of this project was not just to make a model that scores well on a benchmark — it was to make a machine that sees, understands, and speaks.

We think we got partway there. The journey continues.

Built using PyTorch, ResNet-50, LSTM, Flickr30k — as part of the Generative AI course (AI4009), Spring 2026, NUCES.

Tags: Deep Learning Computer Vision NLP Image Captioning PyTorch LSTM Seq2Seq Generative AI Machine Learning ResNet