Can Gemma 4 Replace My 45MB Model? I Benchmarked ResNet18, CLIP, and GPT-4o on 71 Herbs. Here’s the Math.

Real benchmarks from a production mobile herb classifier — why I use a tiny model instead of a 9-billion parameter one, and why CLIP’s 43.7% accuracy is still useful.

Part 5 of 5 in the PlantSnap series — building a production herb classifier for mobile

Everyone asks the same question when I show them PlantSnap:

“Why don’t you just use GPT-4o? Or Gemma? Or Claude?”

Fair question. These models can describe images in stunning detail. They can identify plants, explain their medicinal properties, and even suggest recipes.

But they can’t run on a smartphone in a forest with no signal. And that’s the entire point.

I benchmarked three approaches — ResNet18 (my production model), CLIP ViT-B/32 (my zero-shot fallback), and the latest foundation models — on the same 71 medicinal herbs. The results tell a story about when bigger isn’t better.

The Architecture — How They All Fit Together

Before the benchmarks, here’s how my system actually works in production:

Three tiers of intelligence, each chosen for a reason.

The Three Contenders

Tier 1 — ResNet18 (My Production Model)

ResNet18 is an 18-layer convolutional neural network originally trained on ImageNet’s 1.4 million images [1]. I fine-tuned it on 5,400 herb images across 71 classes using PyTorch, with GPU acceleration on my laptop.

The key insight from my training runs: unfreezing just the final residual block (layer4) and using discriminative learning rates — 10x lower for layer4 than for the classification head — gained me 16 percentage points of accuracy. One line of code (param.requires_grad = True) was responsible for the biggest improvement in the entire project.

The model exports to an on-device ML framework and runs on the phone’s dedicated ML accelerator at 62ms per inference. At 45MB, it fits comfortably on any modern smartphone.

Model size:    45 MB
Parameters:    11 million
Inference:     62ms (on-device ML accelerator)
Accuracy:      62.7% (71 classes)
Runs offline:  Yes ✅
Training:      5,400 images, 5 runs

Tier 2 — CLIP ViT-B/32 (My Zero-Shot Fallback)

CLIP is a contrastive model that maps images and text into the same 512-dimensional embedding space [2]. It was trained on 400 million image-text pairs from the internet — not on my herbs.

That’s the magic and the limitation. CLIP has never seen my specific herb training data, yet it can match a photo of lavender to the text “lavender” through learned visual-language alignment. This is called zero-shot classification.

I encode all 71 herb names as text vectors once at startup, then compare each query image to all 71 vectors using cosine similarity. The highest similarity wins.

Model size:    338 MB
Parameters:    151 million
Inference:     ~60ms (after model load)
Accuracy:      43.7% (71 classes, zero-shot)
Runs offline:  Server only (too large for mobile)
Training:      None needed — zero-shot

Tier 3 — Foundation Models (GPT-4o, Gemma 4, Claude)

These are the models everyone asks about. Gemma 4 has 9 billion parameters [4]. GPT-4o has an estimated 200+ billion. Claude runs at similar scale.

They’re brilliant at describing what they see. Show GPT-4o a chamomile photo and it’ll tell you the species, the family (Asteraceae), the medicinal properties, and whether it’s safe to consume. The quality of understanding is genuinely impressive.

But here’s the math that kills the idea for on-device deployment:

Model size:    4–18 GB (Gemma 4)
               200+ GB (GPT-4o)
Parameters:    2B–9B (Gemma 4)
               200B+ (GPT-4o)
Inference:     500ms–2s (cloud API)
Runs offline:  No ❌
Cost:          $0.005–$0.02 per image
Phone RAM:     6–8 GB total (shared with the OS)

Gemma 4 at 9 billion parameters needs roughly 18 GB of RAM at full precision. A modern flagship phone has 6–8 GB total — shared with the OS, the camera pipeline, and every other running app. The model literally doesn’t fit.

Even quantised to INT4 (roughly 4.5 GB), you’re consuming more than half the device’s total memory for a single model. The phone would kill your app before it completes a single inference.

The Benchmark — 71 Herbs, Three Approaches

I ran CLIP on one representative image from each of my 71 herb classes. Here’s what happened:

CLIP Results: 31/71 correct (43.7%)

What CLIP got right — visually distinctive herbs:

✅ basil, lavender, garlic, echinacea, dandelion,
   calendula, thyme, oregano, feverfew, comfrey,
   reishi, shiitake, black walnut, chickweed,
   california poppy, lemon balm, holy basil tulsi,
   skullcap, vervain, st. john's wort, saw palmetto,
   red clover, nutmeg, motherwort, ginger root,
   coriander, black cohosh, burdock, spilanthes,
   asian ginseng

What CLIP confused — similar-looking herbs:

❌ chamomile      → predicted feverfew    (white flowers, yellow center)
❌ peppermint     → predicted lemon balm  (green mint-like leaves)
❌ rosemary       → predicted vervain     (thin needle-like leaves)
❌ sage           → predicted vervain     (gray-green leaves)
❌ ginger         → predicted ginger root (same plant, naming mismatch!)
❌ tulsi          → predicted holy basil  (same plant, different folder name!)

Two of those “errors” are actually naming mismatches — CLIP identified the correct plant but my folder name didn’t match. Adjusted accuracy: roughly 46%.

ResNet18 Results: 62.7% on validation set

ResNet18 outperforms CLIP by 19 points because it was specifically trained on these 71 herbs. It learned the subtle differences between chamomile and feverfew from thousands of labelled examples. CLIP never had that advantage.

Foundation Model Comparison

I didn’t formally benchmark GPT-4o on all 71 herbs (at $0.01 per image, that’s $0.71 per run — manageable, but the latency makes it impractical for a real-time app). However, from informal testing:

GPT-4o describes herbs accurately but:
  - Returns prose, not a class label
  - 500ms–2s per request
  - Requires internet
  - Costs money per request
  - Non-deterministic output

Gemma 4:
  - Similar quality to GPT-4o
  - Smaller (9B params) but still 18 GB
  - Cannot run on a phone ❌
  - Could run on a desktop/server

The Complete Comparison

Why I Don’t “Just Use GPT-4o”

The answer isn’t about accuracy. GPT-4o would probably beat all three on raw herb identification. The answer is about constraints:

1. Offline-first is non-negotiable. PlantSnap users are in forests, mountains, and remote trails. No signal means no API call. A model that requires internet is a model that doesn’t work where people need it most.

2. Latency matters for camera experience. 62ms feels instant — users point the camera and see a result. 500ms–2s creates a loading spinner. For a camera-based identification app, the difference between “instant” and “wait for it” is the difference between useful and annoying.

3. Cost at scale is real. At $0.01 per GPT-4o request, 1,000 daily users making 10 identifications each = $100/day = $36,500/year. ResNet18 on-device costs exactly $0.00 per inference, forever.

4. Privacy on-device. Photos of what someone forages, where they hike, what they consume — this is sensitive data. On-device inference means the image never leaves the phone. No server logs, no API calls, no data retention policies to worry about.

Where the Industry Is Heading

The interesting space is between my current solution and the foundation model dream:

2024: ResNet/EfficientNet on device        ← I am here
2025: Tiny VLMs on edge (~1B params)
2026: On-device multimodal models emerging
2027: Foundation-model-scale on phone? Maybe.

Key models to watch:

- On-device foundation models from major vendors
- Phi-4 Mini (~3.8B, edge-optimised) [3]
- SmolVLM (~2B, designed for edge) [6]
- Gemma Nano (mobile-optimised) [4]
- PaliGemma (vision-language, smaller) [5]

When edge VLMs become compatible with on-device ML frameworks and small enough for a phone’s dedicated ML accelerator, I can swap my CLIP server fallback for an on-device VLM — without changing the mobile architecture. The confidence-threshold pattern (ResNet → fallback) stays the same. Only the fallback model changes.

That’s why architecture matters more than any single model choice.

The Hybrid Architecture — Why Both Models

Neither ResNet18 nor CLIP is sufficient alone. Together, they cover each other’s weaknesses:

ResNet18 (fine-tuned specialist):
  ✅ Fast (62ms)
  ✅ Accurate on known herbs (62.7%)
  ✅ Works offline
  ❌ Can't identify herbs outside its 71 classes
  ❌ Just a label — no explanation

CLIP (zero-shot generalist):
  ✅ No training needed
  ✅ Works on ANY herb, even unseen ones
  ✅ Returns similarity scores (interpretable)
  ❌ Lower accuracy (43.7%)
  ❌ Confuses similar-looking herbs
  ❌ Requires server (too large for phone)

The handoff point is confidence. If ResNet18 is above 50% confident, its fine-tuned knowledge is more reliable than CLIP’s zero-shot guess. Below 50%, ResNet is essentially guessing — and CLIP’s broader knowledge becomes the better bet.

Image of lavender:
  ResNet: "lavender" (92% confidence) → use this ✅

Image of an unusual herb:
  ResNet: "feverfew" (31% confidence) → not sure...
  CLIP:   "calendula" (0.28 similarity) → might be better
  → Show both, flag as "uncertain"

What CLIP Taught Me About Zero-Shot Learning

Running CLIP on 71 herbs revealed something I didn’t expect: the failures are predictable and systematic.

CLIP struggles with herbs that look similar to humans too. Chamomile and feverfew confuse botanists. Peppermint and lemon balm are genuinely hard to tell apart from a photo. CLIP’s confusion matrix mirrors human confusion patterns — it fails where the visual signal is weakest.

This means CLIP’s accuracy would improve significantly with better text prompts. Instead of comparing images to “chamomile,” I could compare to “chamomile flower with white petals and prominent yellow disk-shaped center, feathery divided leaves.” Richer text descriptions give CLIP more to work with.

That’s a future optimisation — prompt engineering for zero-shot classification. Same model, no retraining, better accuracy through better language.

The Three Lessons

1. On-device inference changes everything. Not because the model is smarter — because it works. In a forest, on a plane, in a tunnel. The best model in the world is useless if it requires an internet connection your user doesn’t have.

2. Zero-shot models are fallbacks, not replacements. CLIP at 43.7% can’t replace ResNet18 at 62.7% for known herbs. But for unknown herbs or uncertain predictions, a 43.7% intelligent guess beats a 31% confident wrong answer.

3. Don’t chase parameters — chase constraints. The question isn’t “what’s the most accurate model?” It’s “what’s the most accurate model that fits my constraints?” 45 MB, 62ms, offline, on-device, no cost per inference. ResNet18 wins that contest convincingly. Gemma 4 wins a different contest that my users can’t participate in.

What’s Next

The next real improvement won’t come from a bigger model. It’ll come from better data.

My CLIP quality gate already filters garbage images from the retraining pipeline. User corrections collected through Store and Forward are building a dataset of real-world herb photos — properly photographed, correctly labelled, from actual field conditions.

Clean data from real users will push ResNet18 accuracy past 70%, then 80%. And when edge VLMs finally fit on a phone’s dedicated ML accelerator, the hybrid architecture is ready to adopt them — no redesign needed.

The feedback loop closes. The model gets smarter from every hike.

References

[1] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR. arXiv

[2] Radford, A. et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. ICML. arXiv

[3] Abdin, M. et al. (2024). Phi-4 Technical Report. arXiv

[4] Team Gemma et al. (2024). Gemma: Open Models Based on Gemini Research and Technology. arXiv

[5] Beyer, L. et al. (2024). PaliGemma: A versatile 3B VLM for transfer. arXiv

[6] Allal, L.B. et al. (2025). SmolVLM: Redefining Small and Efficient Multimodal Models. arXiv

Links

🌿 Live API: https://computer-vision-yin8.onrender.com/docs

📱 Article 1: I Trained ResNet18 to Identify 70 Medicinal Herbs

📱 Article 2: I Built an ML App for Foragers in Forests. Server-Side Inference Would Have Killed It.

📱 Article 3: I Ran My Herb Classifier 4 Times. The Third Run Made It Worse. Here’s Why.

📱 Article 4: I Built a Production ML Feedback Loop with FastAPI and Render. Here’s What Actually Broke.

💻 GitHub: https://github.com/rachana1101/plantsnap 💼 LinkedIn: https://www.linkedin.com/in/rachana1101/

If this resonated, follow for upcoming Articles.

Building something similar? I’d love to hear about it — drop a comment or connect on LinkedIn!

PlantSnap and HerbCam are personal side projects built on my own time and resources. All views are my own and unrelated to any employer.