The Need for v2: What Was Wrong with gUrrT-v1

From the questions that have been coming in, another prominent one is Why v2? Are you doing anything new or is it just some minor changes being shipped?

The answer is we are revamping everything. And by everything I mean EVERYTHING. THE ENTIRE CODEBASE.

To understand why v2, we first need to look at what was wrong with v1.

Quick Recap

If someone is reading for the first time Video Language Models make video query-able. They take a video, extract context from it, and hand it to an LLM, which with its existing knowledge base presents answers to your queries. More on that here → Does gUrrT have a benchmark to compete against?

Now gUrrT works on the sweet spot of context gathering gathering the key content that contains the maximum amount of information about the idea.

Since gUrrT is majorly built for understanding online lectures, that specifically means gathering the correct slides or frames where actual content changes.

Almost all educative videos on the internet involve a partially or fully visible person and a slide or a blackboard. If you want to understand the lecture, all the correct information is either in the slide frames, the blackboard frames, or what is being said by the instructor.

For a moment, let us overlook the voice of the instructor.

The Frame Problem

A video is a culmination of frames.

• A 10 min video at 30fps → 18,000 frames
• A 30 min video → 54,000 frames
• A 1 hour video → 1,08,000 frames

We do not actually need all of these. That is just too much redundant information. In an educative video it is highly unlikely that something changes between the 30 frames of a second.

The frames that contain actual unique information are only those where a new slide appears or something new has been written.

If a person is teaching with slides the frame where the new slide appears with all its content is the useful one. The number of useful frames is more or less equal to the number of actual slides the instructor used.

If a person is using a blackboard the frame right before they rub it off, when everything is written and visible, is the one that matters.

This is the entire focus. Extract highly relevant data. Slide changes. Actual information change. Processing every frame would be computationally disastrous.

The Problem with v1

gUrrT v1 had two frame extraction algorithms. Both failed in different ways.

1) Scene Detection

The content-aware scene detector detects jump cuts in the input video. Internally it converts each frame from RGB into HSV, takes the average difference across all channels from frame to frame, and when this exceeds a set threshold, a scene change is triggered.

The idea was for each detected scene, extract 3 keyframes: start, middle, and end, to capture the overall information about that scene.

The assumption behind this was that slide changes would correspond to scene changes.

If you think about it, that is not true.

In a movie there are camera cuts, dramatic color changes, completely different scenes. In an educative video there is none of that. It is just a stale blackboard whose content gets rewritten with the same chalk, or a slide that gets changed the change appears so minor that most of the time the algorithm skips right past it. What it does trigger on is the instructor walking or waving their hands, which is actual motion, actual color shift frame to frame.

So most of the time the algorithm would detect zero real scene changes, and I would have to fall back on the second approach.

2) Uniform Frame Sampling

Simply sampling the video at 1 frame per second every 30th frame.

This is not unique to gUrrT v1. A large portion of the video understanding space defaults to this exact strategy, especially models working under compute constraints.

• LLaVA-NeXT-Video uniformly samples 32 frames per clip
• Video-LLaMA uses 8 uniformly sampled frames
• SmolVLM2 samples 50 frames per video
• MiniCPM-V 4.6 one of the most efficient mobile VLMs today is capped at 128 frames

They all hit the same ceiling eventually.

gUrrT-v1: Key Frame Extraction Workflow

For small videos, UFS is fine. 2 minutes gives 3,600 frames, you take 120. 4 minutes gives 7,200, you take 240. But as video length increases, the number of frames being processed increases linearly and on 4 to 6 GB of consumer grade VRAM, inference time for those frames becomes the real problem.

What the Data Actually Looked Like

Here is what v1 produced in practice across videos of different lengths.

gUrrT-v1: Key Frame Extraction Data

Look at the longer videos. Scene detection returned 2, 5, and 6 frames for videos between 23 minutes and 1 hour. Completely blind. The algorithm could not see the slides changing at all.

Look at the short video. 105 frames for a 1 min 45 sec video all of them noise, all of them uniform samples of an instructor standing in front of a slide that barely changed.

The Core Failure

When scene detection works, it is fast but all it captures is noise. Instructor motion. Garbage content.

When UFS works, it captures the video throughout but it captures redundant frames, noisy frames, the instructor repeated over and over. Educative videos do not have a lot of actual content changes. That is the point. So UFS just keeps collecting the same information again and again.

The backbone is distraught.

What v2 Does About It

This is what gUrrT v2 solves entirely.

NO CAPTURING OF NOISY MOTION. NO FRAME REPEATED OVER AND OVER. PURE SLIDE CHANGES. ACTUAL INFORMATION CHANGE.

I will not say it is perfect. But it is 100x better than what was done in v1. It is 4x faster.

Below is a Drive link the frames gUrrT v1 extracted vs the frames gUrrT v2 extracted, alongside the actual videos they were run on. See it for yourself.

https://drive.google.com/drive/folders/1eoZiQ8pmfhcb3MjXOFlCuUc-rmvQ7eDR?usp=sharing

v2 RELEASING SOON.