Built the Video Editor Behind My Automated History Channel

Here’s How the Footage Actually Gets Made

A companion piece to “I Automated a YouTube Shorts History Channel — and Here’s What the Data Taught Me.” Last time I told you what the numbers said. This time I’m opening the hood on the machine that makes the videos.

In the last article I showed you the analytics: 322 Shorts, a 4.8k-view best day, a 0-view worst day, and the brutal lesson that if Scene 1 bleeds viewers the video is already dead. A lot of you wrote back with the same question, more or less politely: “Okay, but who actually edits the videos?”

Nobody does. That’s the point.

I’m writing this now because, after a year and change, I’m winding the channel down. Not because it failed, but because I got what I came for and my curiosity has wandered somewhere new (more on that at the end). So consider this both a how-it-works teardown and a bit of a send-off. The engine outlived the channel, which is exactly how it should be.

There’s no human dragging clips around a timeline, no After Effects project, no “let me just nudge this caption two frames.” A folder of images and voiceovers goes in one end, and a finished, captioned, music-scored, YouTube-ready MP4 comes out the other, encoded on a GPU and uploaded while I’m at basketball practice. The thing doing that work is a few thousand lines of Python I wrote, the engine behind BlueVideo.ai. This is the story of how it works, and how you could build your own.

Six things I’d tell my past self

1. Treat the edit as a function, not a craft. folder → mp4. Everything else is implementation detail. Once you frame it that way, the whole "I can't automate video, it's too creative" mental block evaporates.
2. FFmpeg is the engine. MoviePy is the steering wheel. Don’t fight FFmpeg; wrap it.
3. The audio is the timeline. I spent weeks thinking in terms of video frames. Wrong. The voiceover is the clock. Everything else hangs off it.
4. Captions don’t need the cloud. A 40 MB offline speech model gives you word-level timing for free, and your script never leaves the box.
5. Hard-coding the GPU encoder will betray you. Probe for it at runtime and fall back. Future-you, running on a different machine, will be grateful.
6. Silence is the enemy of retention. The single highest-ROI line of code I wrote compresses long pauses. Pacing is editing.

Why write my own editor at all?

I’ll be honest: I tried not to. I looked at the SaaS “AI video” tools. They’re fine if you want what everyone else has. But I was shipping a video every three hours, in multiple languages, with a specific look, on a budget of roughly zero dollars per render. I needed something that ran on hardware I already owned and never sent me a per-export invoice.

So the requirements were blunt:

• No human in the loop. One API call per video. Fire and forget.
• Cheap to run. My machine, my GPU, my electricity.
• Opinionated. Same motion language, same caption style, every time.
• Resilient. It runs unattended for days. It cannot fall over because one voiceover was secretly a .wav wearing an .mp3 costume. (More on that later; it's a real bug and it's funny.)

The result is a Python service. One Flask endpoint, POST /process, takes a Google Drive folder ID and does the rest.

The garage stack (editing edition)

This is the bill of materials for the editing engine specifically, separate from the channel-orchestration n8n stuff I covered last time:

Orchestration is Python + Flask, served by Gunicorn: one endpoint, dead simple. Timeline composition runs on MoviePy, which lets me describe an edit in code. The actual heavy lifting (encode, decode, probe) is FFmpeg / FFprobe, the workhorse of the whole thing. GPU encoding goes through NVIDIA NVENC (h264_nvenc), so a render that takes minutes on CPU takes seconds.

For image smarts and effects (saliency, zoom, parallax) I lean on OpenCV + NumPy. Caption timing comes from Vosk (offline speech-to-text), which gives me word-level timestamps with no API bill. Publishing hits the Google Drive / YouTube / TikTok APIs, uploading to all three from one call. And the whole thing is packaged with Docker (CUDA base) plus systemd, so it runs unattended and restarts itself.

Notice what’s not there: any cloud video API, any paid render farm, any TTS. The voiceovers are generated upstream (that’s the AI crew from the last article) and dropped into the folder. BlueVideo.ai just consumes them.

The contract: one JSON file to rule the edit

Everything keys off a single story.json. If you take one idea from this article, take this one: define the contract first, build everything else against it.

{
  "title": "The Day Queen Victoria Survived an Assassination",
  "description": "On June 13th, 1842... #history #shorts",
  "video_type": "short",
  "background_music": "elegy.mp3",
  "text_overlay": true,
  "illustrations": [
    { "voice_over": "London. A spring afternoon in 1842.", "effect": "saliency_zoom_in" },
    { "voice_over": "A young queen rides out in an open carriage." }
  ]
}

Each entry in illustrations is a scene: a line of narration, an image (or a short clip), and optionally an effect. The folder next to it holds image_leo_000.jpg, voiceover_000.mp3, and so on. That's the whole input.

What actually happens to the footage

When the call comes in, the pipeline runs eight stages. Each one maps to a single Python module, which is deliberate: when something breaks at 6 a.m. I want to know exactly which file to open.

1. Ingest

Download the images, voiceovers, and any motion clips from Drive. Parse the JSON. Nothing clever.

2. Clean the audio (this is where the magic is)

This is the stage I underestimated the most. The audio coming out of a TTS engine is technically fine and experientially flabby. Three fixes:

• Normalize the format by sniffing, not by trusting. I learned the hard way that a file named voiceover.mp3 is sometimes raw WAV data, or PCM, or whatever the upstream tool felt like that day. So I probe the actual codec with ffprobe and ignore the extension entirely. Never trust a file extension. It will lie to you.
• Trim trailing silence. TTS loves to leave a beat of dead air at the end of a line.
• Compress long pauses. Any silence longer than two seconds gets clamped to exactly two seconds. I do this with a NumPy scan over the raw 16-bit samples, measuring loudness and squashing the dead runs.

That last one sounds trivial. It was the biggest single retention win in the whole system. Dead air is where viewers leave. Pacing is editing.

3. Time the captions, offline

More people should steal this trick. I run the cleaned voiceover through Vosk, a small offline speech-recognition model, and get back every word with a start and end timestamp:

[ {"word": "London",  "start": 0.12, "end": 0.61},
  {"word": "a",       "start": 0.61, "end": 0.69},
  {"word": "spring",  "start": 0.69, "end": 1.04} ]

Now I can drop captions on screen exactly when each word is spoken. No cloud transcription bill, no rate limits, and (the part my security-brain cares about) the script never leaves the machine. For a channel pumping out videos every few hours, “free and local” beats “slightly more accurate but metered” every time.

4. Make the stills move

A static image held for eight seconds is death on Shorts. So every still gets motion. The default is a saliency zoom: OpenCV computes which part of the image is visually “loud” (a spectral-residual saliency map: basically, where would your eye go first?), and I do a slow Ken-Burns push toward that point. The viewer’s eye gets led exactly where the composition wants it, automatically.

If a scene wants something fancier, the effects library has depth-map parallax, a color-bloom reveal, a micro-glitch emphasis, a typewriter text reveal, and a few others. But honestly, 80% of scenes are just a well-aimed zoom. Restraint reads as professional.

5. Burn in the captions

Bold text, fat black outline, lower third. Synced to the Vosk timings from step 3. Optional title card up top.

6. Score it

Background music, looped to fit the scene and ducked to ~50% so it sits under the voice. Consecutive scenes that share a track don’t restart the music on every cut, a small thing that makes the whole video feel composed rather than stitched.

7. Concatenate and encode

All the scenes get joined, then encoded to H.264: 24 fps, ~4 Mbps, +faststart so it starts playing before it's fully buffered. The encoder picks itself from a fallback chain:

h264_nvenc  (GPU, fast)
   ↓ not available?
h264_v4l2m2m  (other hardware)
   ↓ not available?
libx264  (software, slow but always works)

This is the line of code that lets the exact same project run on my GPU box and on a laptop without changing anything. Probe, don’t assume.

8. Publish

Upload to Drive, then to YouTube as unlisted first (so the system can add a pinned comment before it goes live), with optional scheduled publishing. If the YouTube quota runs out mid-batch, it rotates to a backup credential and keeps going. Optional TikTok upload. Then it returns the links and goes back to sleep.

Everything it does, in one list

If I had to pitch BlueVideo.ai to a colleague over coffee, this is the napkin version, the features that actually earn their keep:

• ✅ Fully automated assembly: a finished video from a single JSON description, no timeline editing.
• ✅ One-call publishing: YouTube, TikTok, and Google Drive from a single API request.
• ✅ Offline auto-captions: synced to narration via Vosk, no cloud cost and no script ever leaving the box.
• ✅ Saliency-aware Ken-Burns zoom: OpenCV finds where the eye lands and pushes toward it, plus 8 visual effects.
• ✅ Audio hygiene: codec normalization by probing, trailing-silence trimming, and long-pause compression.
• ✅ Background-music mixing: looped and ducked under the voice, with per-scene track overrides.
• ✅ Vertical and horizontal output: Shorts/TikTok or standard, same pipeline.
• ✅ GPU-accelerated encode: h264_nvenc when it's there, graceful software fallback when it isn't.
• ✅ Multi-language localization: translated captions and a dedicated endpoint for localized audio export.
• ✅ Resilient publishing: OAuth auto-refresh and YouTube quota-project rotation so a long batch doesn’t stall.

The bug that taught me the most

Early on, roughly one video in twenty would come out silent or garbled. Maddening, because the input “looked” fine. The culprit: upstream tools were handing me audio with the wrong file extension. WAV bytes in an .mp3, raw PCM in something that claimed to be encoded. MoviePy would believe the extension, hand it to FFmpeg with the wrong assumptions, and FFmpeg would produce noise or nothing.

The fix wasn’t clever code. It was humility: stop trusting metadata, probe the actual bytes, and re-encode to a known-good format before doing anything else. Half my “AI video editing” system is really just defensive plumbing around the reality that real-world files are a mess. That’s not a footnote. That’s the job.

Want to build your own? Here’s the playbook

People assume you need a team to build something like this. You need an AI coding assistant and a weekend of patience. I built and rebuilt most of BlueVideo.ai with Claude Code and Codex riding shotgun. The trick is to never ask for “an AI video editor” up front; you’ll get a tangled mess. Build it the way the pipeline runs: one stage at a time, each one tested before you move on.

A sequence that works:

1. Set the guardrails. Tell the assistant the stack up front: “Python + Flask, MoviePy for composition, FFmpeg with an h264_nvenc→libx264 fallback, Vosk for offline word timing, OpenCV for effects. One module per concern. Probe codecs with ffprobe; never trust file extensions."
2. Define the contract. Have it write the story.json schema and a loader first. Everything hangs off this.
3. Build modules one at a time, with tests. For example: “Write audio_utils.py: normalize any input to mono PCM WAV using ffprobe + ffmpeg, trim trailing silence, and compress silence runs longer than 2s to exactly 2s with a NumPy scan over int16 samples. Add pytest tests using synthetic WAVs." Then the saliency module. Then the Vosk timing. Then assembly.
4. Wire the API last on the build, first on the test. A single POST /process plus a GET /health.
5. Add publishing and OAuth at the very end. It needs real credentials and it’s the fiddliest part.
6. Containerize it so it runs unattended.

Two hard-won tips for working with the assistant. First, give it a real sample folder (one JSON, two images, two short voiceovers) so it can run the thing end to end while it iterates, instead of guessing. Second, make it run the tests after every module before moving on. The discipline of “green before next” is what keeps an AI-built codebase from quietly rotting.

Closing thoughts, and what’s next

The first article was about what the audience taught me. This one is about what the machine taught me, and it comes down to something almost boring: editing, automated, is mostly good defaults applied relentlessly. Aim the zoom where the eye already wants to go. Kill the dead air. Sync the words. Duck the music. Encode on whatever hardware you’ve got. Do it the same way every time, ten thousand times, without getting tired.

I didn’t replace an editor with AI. I encoded an editor’s judgment into a function and let it run while I coached basketball. That’s the whole trick: not full automation, but taste, written down once and executed forever.

So why am I winding it down? Because the build taught me what I actually wanted to know, that one person can stand up a fully automated media pipeline on hardware they already own, and a year of running it scratched the itch. The history-Shorts subject was always the excuse; the pipeline was the project. And the pipeline doesn’t die with the channel. I’m pointing the exact same playbook (script, voice, assemble, encode, publish, all unattended) at the topics I actually read about for fun: AI, humanoid robotics, and fusion energy, and wrote a book about it, learn more at **unscarcity.ai**.

That’s a podcast now, **Minds, Bodies, and Terawatts**, where I track the real-time collision of those three forces as the news breaks. Same automation DNA, new medium: audio instead of vertical video, ideas instead of dates-in-history. If the back half of this article was the part you nodded along to, that’s the project to follow next.

If you build your own version of any of this, I’d genuinely love to see it. The channel is going quiet, but the GPU in the garage is still humming. It just has a new job now.

Next time someone asks who edits my videos, I’ll send them this. Next time they ask what I’m building, I’ll send them the podcast.

Built with Python, FFmpeg, MoviePy, OpenCV, and Vosk. Coded alongside Claude Code and Codex. Running on a GPU box in my garage, now pointed at AI, robots, and fusion over at Minds, Bodies, and Terawatts.