Building High-Quality Video Datasets for GenAI: Scoring, Filtering & Deduplication at Scale

When training or fine-tuning generative AI models on video, the quality of your dataset is everything. A clean, diverse, and deduplicated dataset helps your model learn better representations and avoid overfitting early on.

This post walks through:

1. Why clean video data matters
2. Frame sampling techniques
3. Deduplication and quality scoring using CLIP
4. Filtering out low-quality frames
5. Captioning strategies
6. Example mini-pipeline in PySpark + Hugging Face

🎯 Why Quality Matters

Garbage in, garbage out. Low-quality videos — whether blurry, redundant, or narrowly scoped — can reduce model generalization and produce hallucinations or overfitting in generation.

Key qualities for production-grade video datasets:

• Diversity: from angles, environments, actors
• Clean frames: sharp, non-watermarked, non-NSFW
• No duplicates: avoids wasting compute and inflating model comfort zones

📸 Frame Sampling Techniques

Triangulating the most informative frames lets you capture relevant info while keeping compute manageable.

• Uniform sampling: pick a frame every N seconds, runs fast but may miss key scenes
• Scene detection: use shot boundary detection to sample at cut points
• Perceptual sampling: rely on visual similarity metrics (e.g. CLIP or SSIM) to pick unique frames

A hybrid strategy — uniform plus scene change detection — often strikes the right balance.

✂️ CLIP for Deduplication & Quality Scoring

Use CLIP embeddings to detect similar frames and assess visual quality.

Deduplication:

• Generate CLIP embeddings for all frames
• Cluster or compute pairwise similarity
• Keep only distinct frames (e.g. cosine similarity < 0.85)

Quality scoring:

• Use image-aesthetic scorers or watermark detectors
• Combine with metrics like contrast, brightness, texture
• Rank frames and choose the top X% to include or discard the rest

This ensures variety and clean visuals for model training.

🚫 Filtering Low-Quality Frames

Automatically exclude frames that are:

• Blurry: low contrast or high variance in Laplacian gradients
• NSFW: using pre-trained filters
• Low entropy: single-color or uniform frames
• Watermarked or OCR-heavy: to avoid copyright and clutter

Filtering early saves resources and improves dataset purity.

📝 Captioning: Making Video Multimodal

To align language and vision, generate accurate, rich captions.

• Whisper: speech-to-text from audio tracks
• BLIP-2: descriptive image captions
• LLaVA: advanced videoscaping with frame sequences
• Multimodal pipelines combine them for tight alignment

Frame-level captions plus subtitles provide deep context for fine-tuning video-language models.

🔧 Mini-Pipeline: PySpark + Hugging Face

from pyspark.sql import SparkSession
from transformers import CLIPProcessor, CLIPModel, Blip2Processor, Blip2ForConditionalGeneration, WhisperProcessor, WhisperForConditionalGeneration

spark = SparkSession.builder.appName('video_ds').getOrCreate()
df = spark.read.parquet('video_frames.parquet')  # columns: video_id, timestamp, image_bytes

clip = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
clip_proc = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')
blip_proc = Blip2Processor.from_pretrained('Salesforce/blip2-flan-t5-xl')
blip = Blip2ForConditionalGeneration.from_pretrained('Salesforce/blip2-flan-t5-xl')
whisper = WhisperForConditionalGeneration.from_pretrained('openai/whisper-small')
whisper_proc = WhisperProcessor.from_pretrained('openai/whisper-small')

def extract_clip_emb(image):
    inputs = clip_proc(images=image, return_tensors='pt')
    return clip.get_image_features(**inputs).detach().tolist()

def caption_frame(image):
    inputs = blip_proc(images=image, return_tensors='pt')
    return blip.generate(**inputs).to_list()[0]

def transcript_audio(audio):
    inputs = whisper_proc(audio, return_tensors='pt', sampling_rate=16000)
    return whisper.generate(**inputs).to_list()[0]

df = df.withColumn('clip_emb', extract_clip_emb('image'))
df = df.repartition('video_id').groupBy('video_id').apply(dedupe_by_clip)
df = df.filter(~is_blurry('image') & ~is_nsfw('image'))
df = df.withColumn('caption', caption_frame('image'))
df.write.parquet('filtered_captioned_video_frames.parquet')

This pipeline:

• Embeds frames in CLIP
• Deduplicates per video
• Filters low-quality or NSFW frames
• Adds BLIP-2 captions
• Outputs a ready-to-train dataset

✅ Wrap-Up

By using metadata-aware sampling, CLIP-based deduplication and scoring, smart filtering, and multimodal captioning, you ensure your video dataset is:

• Clean and informative
• Diverse in content
• Rich in alignment between image and text

Such data quality unlocks better generalization and performance in GenAI video models.