Fruits Decay Prediction using CONVLSTM and EfficientB3 with Additional CLIP Pipeline

I’ve always felt annoyed when fruits on my dining table somehow got ripe even before I touched them. This is the first idea that came to my mind. After a little bit of research, it turns out it’s possible with ConvoLSTM (and image processing). My goal is to make it easier for us to know when fruit will go bad just by taking a photo. In the process, there is so much improvement and change. I developed a model to predict the shelf life of fruits such as avocados, bananas, pears, and tomatoes (i only managed 4 fruits because the dataset collection process is tuff). I really think that this project can be improved so much better, but yeah, here is what I got so far.

What will this program do?

Here's how it goes:

You snap a photo -> The AI checks if it’s a valid fruit (avocado, banana, pear, tomato) :

• If it’s a foreign object -> identifies what the object actually is, and safely rejects it.
• If it is a valid fruit (banana, avocado, pear, tomato) -> It saves the data -> Once it has a few days of photos (minimum of 3), it predicts how many days until the fruit rots.

Mainly, divided into three parts :

How to distinguish fruits to other foods?

Yep, used Open Set Recognition and EfficientNetB3

Traditional image classifiers are often too confident

If you show them a shoe, they might just guess it’s a weird-looking avocado. OSR prevents this by measuring “energy.” If an image’s energy is weirdly high, the model knows the object is Out-of-Distribution (OOD) (meaning the model can’t recognize it)

We run the image through the EfficientNet-B3 backbone. Instead of just looking at the final probability score, we calculate the Free Energy. We set a specific threshold during training.

Tracking the rot

with ConvLSTM of course

the OSR already stripped the image down to a 2D feature map (in the notebook). We retrieve a sequence of these saved spatial feature maps over a window of a few consecutive days. Then, we feed this sequence of tensors into the ConvLSTM, which tracks daily changes to estimate remaining shelf life using regression.

How does it exactly calculate the days left?

Here is the secret sauce: metadata.json.

The ConvLSTM doesn't magically know the total lifespan of every fruit. It analyzes the sequence to predict decay progress (e.g., "This banana's visual changes suggest it's rotting on Day 4").

{

“spoil_day” : n

}

To get the remaining days, the system checks our metadata.json file, which acts as a biological dictionary storing the maximum absolute shelf life (spoil_day) for each fruit. Then, it performs a simple math equation:

Spoil Day (from metadata) - Current Decay Progress (from ConvLSTM) = Remaining Shelf Life.

Wait, is that a cat?

When the OSR rejects an item, simply throwing an “Unknown Error” is bad user experience. We want the system to be smart enough to tell the user what foreign object they actually uploaded.

If OSR flags an image as OOD, we route it to OpenAI’s Zero-Shot CLIP model. We compare the image against a file containing around 500 items. CLIP finds the closest text match.

Basically, you don't need to retrain the model for every new image detected.

Here is the thing!

The model can’t predict if there is only one image, for example, an image on day 1. Because we need a sequence of historical data to track the physical rotting trend over time. So, I came up with using a database, but it’s only for keeping track of the daily feature tensors and session IDs locally with SQLite3 (cuz its incredibly lightweight and serverless).

Datasets

There are two datasets i used. First is for the osr second is for convolstm. For the OSR, I mixed standard fruit datasets with random objects.

For the temporal dataset, it was a manual grind… (I take 7–11 photos of banana, avocado, tomato, and pear daily). Literally watched them rot. I took pictures every single day under consistent lighting and angles to capture their gradual degradation accurately.

If you want to see the temporal dataset, go visit my github repo: NaufalSyahputra9/food-decay-predictor

Training

The OSR was trained using an Energy-based margin loss to push knowns and unknowns apart. The ConvLSTM was trained as a regression task using Mean Squared Error (MSE) loss and the AdamW optimizer.

Because the temporal dataset was tiny (collecting rotting fruit data takes days), I used K-Fold Cross-Validation to ensure the model wasn’t just memorizing specific fruits. I also applied Data Augmentation to auto-generate thousands of extra image variations (flips, rotations, color tweaks) from the original photos. (even though it makes the training process longer).

Evaluation

Most data points closely follow the diagonal line. But still, it’s not 100% perfect.The outliers primarily occur in the middle of the timeline, as the AI readily identifies perfectly fresh or completely rotten fruit.

During transition days, subtle visual cues can mislead the AI. Minor lighting inconsistencies or internal decay masked by fresh-looking skin can sometimes cause errors.

The model understands that shelf life decreases over time and differentiates between fruits based on their initial spoil dates.

If you look closely, the lines aren’t perfectly straight or smooth! There are moments where the line goes flat for a day or two, and then suddenly drops sharply. This is a significant result because the model predicts based on visual changes alone.

A banana's yellow appearance might remain constant for days, but it can suddenly develop brown spots overnight. The ConvLSTM effectively captured this real-world, non-linear rotting behavior.

Closing

What an exhausting process, though I learned a lot. Building an end-to-end ML pipeline from manual data collection to a Dockerized full-stack deployment was a massive challenge.

Hopefully, I can make many improvements. Thank youu!