The navigation menu for my garden inventory app

Building a Plant Collection Inventory

Using my DGX Spark to catalogue my plant collection.

A Nerd in the Garden

Spending time with plants in the real, messy, organic world is a way for me to switch off from the intensity of work. But at heart I am still a nerd, and as such my love of data seeps into my gardening. I also have a tendency for collection (conifers, bonsai, Australian plants) and so I wanted to have some metrics to better understand the scope and scale of my plant collection. I thought it would be interesting to look at some metrics around the collection which reflect biodiversity and could help identify where to expand it.

This has previously been a daunting prospect because we have a LOT of plants. And since I dabble in bonsai, I have small trees in pots wherever space allows. As any plant-lover knows, there is always room for one more! But there was no way I was going to go around and do manual data entry to create an inventory — boring process work completely contradicts the entire point of being in the garden.

The Gift of Garden Leave

Back in March I quit my role at AWS to take up a new position elsewhere. After some weeks of being invited to progressively fewer and fewer meetings, eventually I was put on garden leave to serve out the rest of my 3 month notice period (for US readers — this is a European thing). This left me with time on my hands to spend with my DGX Spark AND in the garden, so it seemed like the ideal opportunity to finally build my plant collection inventory.

I also invested in a Claude Max subscription for a couple of months so could use it for some serious vibe coding.

What I Wanted

I wanted to know the breadth and depth of my plant collection including botanical families, genera, species and number of plants, as well as representation across regions & families. I also wanted to see how my collection compares from a biodiversity and scale point of view to other gardens or smaller botanical gardens.

Some key requirements included:

1. Ingesting videos with narration as the primary input
2. Identifying plants from whatever information I knew about them — ranging from the latin name through common name all the way to nothing at all
3. CRUD operations for tweaking the details
4. Full local operation on my DGX Spark

The Benefit of DGX Spark for this Project

There were two reasons I wanted to build a completely local solution.

Firstly, I have plants everywhere on my property and the videos are taken in my house and garden — I prefer to keep these local to my machine. Not just that, but sending a full video off for a cloud-based model to analyse would expose images of my house to that model and send them across the internet. So local gives me complete privacy.

Secondly, I invested $4400 (including tax) in my DGX Spark so ideally I should take advantage of that sunk cost (aka my 128GB unified memory). Gemma-4B-26B-A4B-it GGUF is around 16GB (Q4) with mmproj file of 1.2GB (required for multimodal operation), and in total uses around 45–50GB when served with llama.cpp. This leaves plenty of memory for Whisper and the app to run. So hosting the AI models locally reduces the cost.

Creating the Data

Like with any AI project, it’s all about the data you have to work with. In this one, a user records a video as they walk around their garden or plant zone. They state the zone name “I’m in the back garden”, and then video each plant while narrating what they know about it. This could be the Latin name “This is a Cephalotaxus” or a common name “Here is a bottle tree” or if they don’t know they can just capture it as is. They can also say “Here are five Scots pines”.

The goal is to provide individual plant observations which line up a description with an image to be parsed when ingested. I also preferred to do shorter videos which could be added together over time. I wanted to allow for manual photo uploads over time as well.

The High Level Architecture

1. Ingest Pipeline

The ingest pipeline is seven sequential steps, four of which run local AI models:

1. Transcribe: the audio is transcribed using OpenAI’s Whisper.
2. Parse: plant mentions are extracted from the audio transcript by the Gemma LLM.
3. Locate: the video frame for each mention is identified and a bounding box drawn around it using the VLM capability of Gemma.
4. Validate: the cropped image is sent to PlantNet for a second opinion identification.
5. Resolve: the spoken text, PlantNet’s answer and the garden’s existing context are fused into ranked identity candidates by the LLM.
6. Write: a plant observation record is created, with either automatic ID (if confidence is high) or flagged for human review.
7. Enrich: Public APIs are used to add further metadata including botanical family, native region and conservation status.

2. Local Models

To run the pipeline the following local AI models are used:

• Whisper large-v3 (~3 GB) — OpenAI’s speech-to-text model, run via the whisper-timestamped library
• Gemma 4 26B-A4B (Q4_K_XL GGUF, 16 GB on disk; 45–50 GB resident with KV cache) — Google’s multimodal model, served by llama.cpp’s llama-server.
• Gemma’s multimodal projector (mmproj-BF16.gguf, 1.2 GB on disk) loaded by the same llama-server process via the — mmproj flag, this is what turns Gemma into a vision-language model.

3. FastAPI Web App

The web app provides the user with the ability to upload videos & photos, view the collected data, update plant records and see the benchmark metrics.

What it Produced

I recorded 24 videos across 8 zones in my house and garden. After ingestion and tweaking, this is the summary dashboard. I’m extremely pleased! We do have a very diverse collection after all, and I finally have a catalogue which I can update as we develop the garden.

In my next post I’ll share a few of the technical decisions and gotchas that I experienced during development.