My First Image Captioning Project: Teaching AI to Describe Pictures

AI4009 — Generative AI Assignment National University of Computer and Emerging Sciences Spring 2026 Submitted by: [Hajra Shehzad, 22f-3443 , fakiha Khalid, 22f-3375, Batch 22, FAST NUCES]

Hey everyone! 👋

I recently completed my first big deep learning assignment — building an image captioning model that looks at a photo and writes a short description in English.

It was exciting (and sometimes frustrating 😅), but I learned so much. In this post, I’ll share what I built, how I did it, and what I learned — in a beginner-friendly way.

What is Image Captioning?

Imagine showing a picture to a friend and asking: “What do you see?”

Your friend says: “A little boy is jumping on a trampoline in the backyard.”

That’s exactly what image captioning does — it teaches a computer to generate natural sentences about what’s in the picture.

This task combines computer vision (understanding images) and natural language processing (generating text).

I used the Flickr30k dataset from Kaggle:

• ~31,783 images
• Each image has 5 different human-written captions
• Total captions: around 159,000

Example: Image → a dog running on the beach Captions:

• “A brown dog is running on the sand near the water.”
• “The dog is playing in the waves.” etc.

Link to dataset: https://www.kaggle.com/datasets/adityajn105/flickr30k

My Model — How It Works (Super Simple Explanation)

I used a classic CNN + RNN approach (very popular for beginners):

1. Encoder (the “eye”): Pre-trained ResNet50 (from torchvision) → takes the image → removes the last classification layer → gives a 2048-dimensional feature vector (a compact summary of the image)

2. Decoder (the “mouth”): LSTM (a type of RNN) → takes the image feature as starting point → generates words one by one (<start> → word1 → word2 → … → <end>)

3. Training: — Teacher forcing (feed correct previous words during training) — Loss: Cross-Entropy (ignores padding tokens) — Optimizer: Adam — Trained for 15 epochs on Kaggle GPU (T4 ×2)

Total learnable parameters: ~11 million

Notebook link: https://github.com/Hajra341/GEN-AI/blob/main/iMageCaptioning.ipynb

Training Journey & Results

Loss started high (~3.6–4.0) and came down nicely:

• Train loss: dropped to ~2.12
• Validation loss: best around ~2.74 (some overfitting after epoch 10)

Here’s the loss curve:

[Insert your loss plot screenshot here]

Some generated captions from test images (beam search with width=3):

[Insert 2–3 screenshots here — choose ones where the caption makes reasonable sense]

Examples:

• Ground truth: “A girl in a red shirt jumps on a trampoline.” Generated: “a girl in a red shirt is jumping in the air .”

• Ground truth: “Two teams are playing football.” Generated: “a soccer game is being played on the field .”

Not perfect yet (repeats “a man in a shirt” sometimes), but for a first project — pretty cool!

BLEU-4 score (on ~800 test images, beam search): Average ≈ [insert your actual number, e.g. 0.214] (This measures how similar generated captions are to human ones — higher is better)

What I Learned (Beginner Lessons)

• Pre-extracting image features with ResNet saves a lot of time
• Vocabulary building & padding captions is tricky but important
• LSTM can generate text, but without attention it forgets details in complex scenes
• Beam search usually gives better captions than greedy (argmax)
• Kaggle notebooks + GPU = free superpower for students!

Future Improvements (If I Continue)

• Add attention (so model focuses on right parts of image)
• Try transformer decoder (like modern models)
• Train longer or use bigger dataset (MSCOCO)
• Add diversity in generation (nucleus/top-k sampling)

Final Words

This project made me realize how powerful (and fun!) multimodal AI can be. From raw images → features → words — it all came together in one notebook.

If you’re a beginner interested in computer vision + NLP, I highly recommend trying image captioning — start small with Flickr8k/Flickr30k and PyTorch.

Thank you to my instructor and friends for support!

GitHub Repo: https://github.com/Hajra341/GEN-AI Live Demo (Gradio): [insert your Gradio public link if you got it working]

Any tips or feedback? Drop a comment below! 😊

MachineLearning #DeepLearning #PyTorch #ImageCaptioning #ComputerVision #GenerativeAI #Beginner