I Built an AI That Turns Any Text Prompt Into a Video — Here’s How (And What Broke Along the Way)

A beginner’s journey from zero to a working AI video pipeline using Gemini, SDXL, Chatterbox, and FFmpeg — all free, all open source, runs on Kaggle.

The Idea

What if you could type a single sentence and get back a complete short-form video?

Not a slideshow. Not a stock footage mashup. A real AI-generated video — with a custom script, AI-generated visuals, and a natural voiceover — ready for YouTube Shorts or Instagram Reels.

That’s exactly what I set out to build. The project is called Prompt2Tube, and after a lot of trial, error, dependency hell, and crashed Colab sessions, I got it working.

Here’s everything I learned — the full pipeline, the mistakes, and how you can build it yourself.

What It Does

You give it a topic. It gives you a video.

Under the hood, six things happen automatically:

Your prompt
    ↓
Gemini writes a scene-by-scene script
    ↓
SDXL generates one AI image per scene
    ↓
Stable Video Diffusion animates each image into a video clip
    ↓
Chatterbox TTS converts narration to audio
    ↓
FFmpeg stitches everything into a final MP4

I tested it with the prompt “A Cat giving Daily Motivation Lecture” and got back a 5-scene video in under 10 minutes on a free Kaggle T4 GPU.

The Stack (All Free, All Open Source)

WhatToolWhy I chose itScript generationGemini 2.5 FlashFree API, structured JSON outputImage generationSDXL via DiffusersRuns on free T4, huge model ecosystemImage to videoStable Video DiffusionAnimates static images into clips, open sourceText-to-speechChatterbox TTSMIT licensed, better than gTTS for narrationVideo renderingFFmpegIndustry standard, completely freeRuntimeKaggle NotebooksFree T4 GPU, stable sessions, built-in secrets

No paid APIs. No subscriptions. Total cost: $0.

Phase 1 — Script Generation with Gemini

The first thing the pipeline does is turn your raw prompt into a structured JSON script — one object per scene, with narration text, an image generation prompt, and a video prompt.

I used Gemini 2.5 Flash because it’s free (no credit card needed), fast, and supports structured JSON output natively — which means no regex parsing or cleanup needed.

The prompt I engineered tells Gemini to think like both a scriptwriter and a visual director:

from google import genai
import json

client = genai.Client(api_key=GEMINI_API_KEY)

def generate_script(user_prompt, num_scenes=5):
    prompt = f"""
You are a YouTube video scriptwriter and visual director.
Create a video script for: "{user_prompt}"

Return ONLY valid JSON — no markdown, no extra text.
Each scene needs: narration, image_prompt, video_prompt, duration_seconds.
"""
    response = client.models.generate_content(
        model="models/gemini-2.5-flash-lite",
        contents=prompt,
        config={"response_mime_type": "application/json"}
    )
    return json.loads(response.text)

What I learned: Ask Gemini to return application/json directly instead of parsing markdown code blocks. It's cleaner, faster, and never breaks.

Phase 2 — Image Generation with SDXL

Once I had the script, I needed one image per scene. I used Stable Diffusion XL (SDXL) via HuggingFace Diffusers.

SDXL runs on a T4 GPU and produces stunning 1344×768 images (16:9 for YouTube, 768×1344 for Shorts).

from diffusers import StableDiffusionXLPipeline, EulerDiscreteScheduler
import torch

pipe = StableDiffusionXLPipeline.from_pretrained(
    'stabilityai/stable-diffusion-xl-base-1.0',
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant='fp16',
).to('cuda')

pipe.enable_xformers_memory_efficient_attention()
pipe.enable_vae_slicing()

For each scene I used the image_prompt from Gemini's output as the SDXL prompt — things like:

“cat professor at podium, dramatic studio lighting, photorealistic, cinematic, 8k”

What I learned: SDXL has specific native resolutions. Stick to them (1024×1024, 1344×768, 768×1344) or you get distorted outputs. Also — always call torch.cuda.empty_cache() between generations or you'll run out of VRAM halfway through.

⚙️ Hardware Config — Tune This For Your Setup

I built this on a free Kaggle T4 (15GB VRAM) so I kept settings conservative. If you have better hardware, here’s how to push quality higher:

# ── HARDWARE CONFIG — change based on your GPU ──────────────────

# Free Kaggle/Colab T4 (15GB) — default, what I used
WIDTH, HEIGHT   = 1344, 768   # 16:9 YouTube
STEPS           = 30          # inference steps — more = better quality
CFG             = 7.5         # guidance scale
USE_REFINER     = False       # refiner needs +4GB VRAM, skip on T4
TORCH_DTYPE     = torch.float16

# RTX 3090 / A100 (24GB+) — higher quality
# WIDTH, HEIGHT = 1536, 640   # cinematic widescreen
# STEPS         = 50          # more steps = sharper detail
# CFG           = 8.0
# USE_REFINER   = True        # enable refiner for extra quality pass
# TORCH_DTYPE   = torch.float16

# A100 80GB / H100 — maximum quality
# WIDTH, HEIGHT = 1536, 640
# STEPS         = 60
# CFG           = 8.5
# USE_REFINER   = True
# TORCH_DTYPE   = torch.bfloat16  # more stable at high res

# ── SPEED vs QUALITY tradeoff ────────────────────────────────────
# preset "ultrafast" → fastest render, lower quality
# preset "fast"      → good balance (what I use)
# preset "slow"      → best quality, takes longer
FFMPEG_PRESET   = "fast"
CRF             = 23          # lower = better quality, bigger file (18-28 range)

If you’re on a free T4, stick with the defaults — they’re optimised for 15GB VRAM. If you’re on a better GPU, uncomment the relevant block and enjoy the quality jump.

Phase 3 — Bringing Images to Life with Stable Video Diffusion

Static images are fine but actual motion makes it feel like a real video. So after SDXL generates each scene image, I pass it through Stable Video Diffusion (SVD) to animate it into a short clip.

The key parameters that affect output quality:

result = pipe(
    image,
    num_frames=32,           # more frames = longer clip, more VRAM
    motion_bucket_id=127,    # 0 = subtle motion, 255 = very dynamic
    noise_aug_strength=0.02, # higher = more creative, less faithful to image
    num_inference_steps=25,  # more steps = smoother motion
)

The motion_bucket_id is the most impactful setting — I found 80–100 works best for calm documentary-style scenes, and 120–150 for more energetic content.

On a T4 I kept resolution low (256×448) to avoid running out of VRAM. If you have a better GPU, push this to 512×896 for a much sharper result.

Phase 4 — Voiceover with Chatterbox TTS

This phase caused the most headaches — and taught me the most.

I first tried ElevenLabs (paid — ruled it out early), then gTTS (completely free but very robotic), then landed on Chatterbox by Resemble AI — open source, MIT licensed, and a clear step up from gTTS. Honestly it still sounds a bit synthetic at times, but for a free POC running entirely on Kaggle it gets the job done. As the project moves forward I’ll be testing paid options for better voice quality.

from chatterbox.tts import ChatterboxTTS
import torchaudio as ta

model = ChatterboxTTS.from_pretrained(device="cuda")

for scene in script['scenes']:
    wav = model.generate(
        scene['narration'],
        exaggeration=0.3,  # calm, natural narration tone
        cfg_weight=0.5,
    )
    ta.save(f'audio/scene_{scene["scene_number"]}.wav', wav, model.sr)

The big lesson here: Chatterbox requires torch==2.6 and transformers==5.2. SDXL works best on torch==2.2 and transformers==4.40. These conflict. Running them in the same environment causes dependency hell.

The solution? Separate environments for each phase. Each phase saves its output to disk. The next phase picks it up. Clean handoff, no conflicts.

Phase 5 — Rendering with FFmpeg

This is where everything comes together. FFmpeg takes each scene’s image + audio and renders them into a video clip, then stitches all clips into the final MP4.

import subprocess, json

def get_audio_duration(audio_path):
    cmd = ['ffprobe', '-v', 'error', '-show_entries',
           'format=duration', '-of', 'json', audio_path]
    result = subprocess.run(cmd, capture_output=True, text=True)
    return float(json.loads(result.stdout)['format']['duration'])

# Render each scene
for i, (img, audio) in enumerate(zip(image_paths, audio_paths), 1):
    duration = get_audio_duration(audio)
    subprocess.run([
        'ffmpeg', '-y',
        '-loop', '1', '-i', img,
        '-i', audio,
        '-t', str(duration),
        '-vf', 'scale=1344:768,format=yuv420p',
        '-c:v', 'libx264', '-preset', 'fast', '-crf', '23',
        '-c:a', 'aac', '-ac', '2', '-ar', '44100',
        '-shortest', f'videos/scene_{i}.mp4'
    ])

What I learned: The Ken Burns zoom effect (zoompan filter in FFmpeg) looks great but is extremely CPU and RAM intensive — it caused my system to hang partway through rendering. Remove it for testing, add it back once everything else works.

What Broke (And How I Fixed It)

Dependency conflicts — Chatterbox, SDXL, and CogVideoX all want different versions of torch and transformers. Solution: separate Kaggle notebooks/environments for each phase, files passed between them.

VRAM running out mid-render — Fixed by calling del pipe, gc.collect(), and torch.cuda.empty_cache() after every phase.

Video hanging mid-render — Caused by loading full MP4 clips into RAM simultaneously. Fixed by pre-trimming clips to audio length before rendering and removing the Ken Burns filter.

Gemini returning markdown instead of JSON — Fixed by setting response_mime_type: application/json in the API config.

Colab session resets — Switched to Kaggle. More stable, better GPU access, cleaner secrets management.

Try It Yourself

The full notebook is open source on GitHub:

🔗 github.com/jill-05/Prompt2Tube

To run it:

1. Open the notebook on Kaggle
2. Enable T4 GPU (Session options → Accelerator → GPU T4 x1)
3. Add your free Gemini API key to Kaggle Secrets
4. Change USER_PROMPT to your topic
5. Run all cells

That’s it. Free GPU, free APIs, working AI video pipeline.

What’s Next

This was just the beginning. The next version is going to be a fully paid, end-to-end production pipeline — better models, better voice, better video quality. I’m currently researching which APIs to use for each phase (Veo 3, Kling, and a few others are on my list) and will share a full breakdown once I’ve tested them.

If you’re interested in following along, star the repo and stay tuned.

Final Thoughts

10 days ago I had no idea how diffusion models worked. I just knew basic Python.

Building Prompt2Tube taught me more about AI, GPU memory management, dependency conflicts, and video rendering than any course I’ve taken — in under two weeks. The best way to learn is to build something that breaks constantly and forces you to figure out why.

If you’re a beginner reading this — start messy. Ship something broken. Fix it. Ship again.

The gap between “I know Python” and “I built an AI video pipeline” is smaller than you think. It took me 10 days. It can take you less.

If this helped you, give it a clap on Medium or share it on LinkedIn. And if you build something with Prompt2Tube, I’d love to see it!

— Connect with me on LinkedIn | Star the repo on GitHub

Tags: #AI #MachineLearning #Python #GenerativeAI #OpenSource #StableDiffusion #TextToVideo #Kaggle #BuildInPublic #MLOps