How I Built a Human-in-the-Loop Multi-Agent Video Studio in 8 Weeks

An 8-week case study in AI-assisted short-form video production — orchestration, review, launch optimization, and human-in-the-loop quality control.

TL;DR — Over eight weeks, a small pet channel(@ryani_n_leo) I run stopped being a channel and became a human-in-the-loop, multi-agent studio that runs itself to produce, measure, and improve four videos a day — with me stepping into every seam to steer direction and quality. Along the way I built 34 agent modules (~25,000 LOC), uploaded 118 videos, made 343 commits across 27 days, and pushed the image-to-video stack from early experiments (sora-2) through Google Veo to BytePlus Seedance 2.0. This is the story of the system, the model journey, and the failures I deliberately kept.

The problem

It started with a false sense of ease. I made character stills with GPT image editing (images.edit) and thought, "oh, this could actually work." And the stills did work — about $0.04 each, one API call. But motion is a completely different, far harder pipeline. A still is one API call; a moving shot is a chain of models, hard constraints, and every failure mode in between.

[Character stills with GPT image editing]

The moment I tried to scale past one-clip-at-a-time, four problems hit at once. The real quality bottleneck was upstream writing, not the video engine. AI-looking footage was ignored by viewers on a young channel. The LLM reviewer waved flawed output straight through. And my Mac infrastructure kept collapsing — in that slow, grinding way where a media library outgrows the disk it lives on. Fixing any one of these alone changed nothing. The system didn’t need a better model. It needed to become an organization.

From stickers to two lanes — when the concept isn’t in the footage

At its core, this channel was about generating endless content from two fixed characters: Ryani, an 11-year-old French bulldog, and Leo, a roughly 8-month-old tabby cat. Everything was a way to keep pulling stories out of those two — seasonal concepts, and sometimes pure flights of imagination.

My original plan was different. I wanted to lay cartoon-cut stickers over real footage — hand-drawn overlays (paws, hearts, labels) on genuine clips. But once I saw firsthand how hard that is to pull off with 100% image-gen or video-gen APIs, I split in a new direction: either edit real footage (RF) or produce it as an AI vtuber (AV).

[Initial Architecture]

At that point the pipeline was simple. A Director agent sets the concept → a Writer agent writes the actual content and captions → a Camera agent finds footage that matches the concept and edits it (or, if no clip exists, generates it and then captions) → finally the PD (a human — me) reviews. Here, the “Director” is just the agent that owns the very first step: deciding the concept. (These four roles later split much finer into the studio below.)

The reason the two lanes exist lives right inside that chain: the concept the Director sets may not exist in any real clip. That’s the wall the Camera agent hits every time — if a matching clip exists, edit it (RF); if not, generate it (AV). Real footage is genuine but bound to what was already filmed: you can’t shoot a scene that never happened. The AI-vtuber lane can generate a concept that was never filmed, but pays for it by having to generate every frame (and most of the failures later in this piece are born right there). So the two lanes aren’t rivals — they’re complementary. RF carries truth; AV carries imagination.

The model journey — sora-2 → Veo → Seedance

Stills were cheap and easy. Motion was neither.

The early i2v experiments started with sora-2. The first thing I learned: the same prompt yields rich motion in one run and a dead-still frame in the next — generation is stochastic. I bolted on logic to auto-check whether a clip had enough motion and retry if it was static; that became the seed of the later “best-of-5.” After A/B/C testing sora-2 against Google Veo, I flipped to Veo in mid-May for stability (Episode 01 was the first finished piece — a 7-clip sora baseline rebuilt as a 5-cut Veo episode). Then, chasing richer, more dynamic motion, I moved again — to BytePlus Seedance 2.0 i2v, which I run in three modes:

• i2v — the standard cut (GPT still → Seedance).
• interp — first/last-frame gap-fill for real footage (up to 4s).
• ref — up to nine reference images to fight character drift across cuts.

Getting to Seedance meant learning exactly where a state-of-the-art video model breaks and building the scaffolding to make it trustworthy. Veo’s first/last-frame interpolation simply wasn’t available to me (every Gemini and Vertex endpoint returned 400/404). Seedance 2.0 can’t mix first/last frames with reference images in one call, and has no scene chaining. So episode continuity is assembled cut by cut (concat + interp gap-fill).

There was also a less glamorous trap that ate a lot of time: every model has a different input-prompt length ceiling. As my system prompts ballooned (the Director context alone ran ~25KB, carrying character sheets and proven motion patterns), calls started dying with opaque API errors. I spent a while treating this as an auth/transport problem, fiddling with endpoints and retries. The real answer was elsewhere — slim the prompt below each model’s ceiling. (Afterward, ephemeral caching cut input cost by up to ~90% on cache hits, which is what made this scale sustainable.) Lesson: a recurring API error isn’t always transport or permissions — first suspect that your input is simply too heavy.

The economics lesson I learned the hard way: a full render is ~$50, and Seedance charges hit my card nearly every day at ~$50 a pop. When I actually summed the billing receipts (my internal cost ledger only captured one week at ~$151 and badly undercounted), the 8-week spend looked like this:

[Cost Table]

Seedance (render) and OpenAI (stills + text) are the costs that attribute cleanly to the pipeline — those two alone are ~$2,700. The rest is fuzzier: Google Cloud (~$955, converted at ₩1,380/$ — beyond Vertex Veo and Gemini, the same account also carries other projects; a single day, May 27, hit ~$362) and Anthropic (one account paying for both the pipeline’s Opus calls and the Claude Code development that built the studio — one sample receipt was $440). Adding those, total provider charges comfortably exceed $4,000, but that figure should be read separately from the pure-pipeline ~$2,700. Either way the conclusion holds: a $50 render is not a $5 iteration. Validate on $0.04 stills first, then pay for motion — a rule I set only after mistaking renders for cheap iteration and burning ~$100 in a single night.

[Weekly Spend Information]

Week by week, the story shows up right in the graph: early on (early/mid May) it’s OpenAI at 100% (stills and text only); late May is Google Cloud at 66–77% (Vertex Veo experiments plus a 3D-modeling detour — the ~$362 day was May 27, though this account mixes in some non-pipeline work, so the share is approximate); and June becomes Seedance at 54–79% — the age of the render. The peak was the week of June 8–14 at ~$891.

Two shape changes — one take to cuts, text to references

Episodes were ~20 seconds. My first instinct was to generate the whole 20 seconds as one take. It broke immediately: a single long call (with 4× slow-motion layered on) produced laggy motion and captions that drifted out of sync with the picture. So an episode split into 4–5 cuts — and, more importantly, cut count stopped being a fixed number and became one cut per story beat (within a readability limit). Continuity across cuts is stitched with concat + interp gap-fill.

The method of generation wandered just as much. I started with text-to-video (pure prompts), but a text prompt invents the location, so the room teleported between cuts. I moved to still → i2v (generate a first frame, then animate it), and increasingly to reference-image-based generation (Seedance’s ref mode, up to nine images). The lesson underneath all of it: when look, set, or props matter, the reference image beats the prompt. Write “photoreal” or “no collar” all you want — whatever the reference shows wins. So the fix was never better wording; it was feeding the right image.

And generation itself was, quite literally, a dice roll. Image gen was; video gen more so — the same prompt and the same reference give a different result every time, and plenty of them are simply unusable. So to land a single cut, I moved to generating five and picking the best (best-of-5). In a stochastic engine, the only way to land one trustworthy frame or cut is to roll several times and choose.

The architecture — one call becomes a studio

[Full Architecture]

The core shift was to stop treating the LLM as one brain and start treating the pipeline as one studio. The four-agent chain above split much finer into a division of labor:

• Writer — Opus, 3 passes (draft → self-critique → revise). Story only: beats, captions, transitions.
• Conte / cue-sheet agent — shot design only: shot size, angle, camera, blocking, first→last delta. It deliberately doesn’t read the captions or prompt strings (a storyboard discipline).
• Director — realizes the cue sheet into a prompt and picks the Seedance mode per cut.
• Cameraman — AV: still → Seedance i2v (best-of-5); RF: clip trim + interp gap-fill.
• Assembly — caption burn-in → normalize + bumpers + BGM.
• Giri (reviewer) — the gate on a finished short: upload / revise / discard. Backed by the deterministic gates below.
• Channel Manager — runs the whole channel by data, not a single cut: title/tag A/B packaging, bandit-driven slot allocation, and portfolio feedback on “which themes and packaging win.”
• Macro Reviewer — judges cross-episode freshness against the latest published videos (no repeats).
• Launch orchestrator + Thompson bandit — schedules 2 AV + 2 RF a day on a lane × timeslot Latin square, measures 48-hour views + retention, and feeds a 3-level Thompson bandit.
• Board Executor — a Slack-based coding agent that edits code, commits, and pushes, under smoke gates, a kill switch, and a shared progress log with my interactive sessions. (It’s orthogonal — it edits the pipeline’s own code and is not in the content flow.)

Ahead of all this sits the concept directive: PD /concept (highest priority) → launch intro → arc season-plan. The timeliness of "what to make" (season, holiday, trend) is driven by arc — not the Channel Manager (a common mix-up). So the review-and-strategy layer is split cleanly across four owners: what to make (timeliness) = arc, how to package and place it = Channel Manager, don't repeat (freshness) = Macro Reviewer, does this one ship = Giri. Lump them into one box and you blur who actually does what — and that distinction is part of the orchestration.

The pipeline runs as a closed loop: publish → analytics (48h) → bandit → the Channel Manager’s portfolio feedback returns to the next batch’s concept directive. Two content lanes (AV, RF) run in parallel, grounded by a three-layer knowledge system (VLM observation / human-authored facts / ask-when-unknown). That knowledge system exists because the pipeline once hallucinated a character trait (it invented a nonexistent “fear of water”) and I never wanted that again.

Orchestration was the bottleneck

The biggest lesson wasn’t about any single agent. It was that the real bottleneck of automation is orchestration — not tuning any one model, but the division of labor and the seams between the parts. No amount of i2v improvement could fix bad upstream writing, so writing had to be isolated into its own 3-pass agent. A blindly-rendering director had to grow an editor’s eye on the actual footage. A rubber-stamping reviewer had to be replaced by signals computed in code. Every hard problem, in the end, was an interface problem.

A fix that won’t stick — contradictions between agents, and rule sedimentation

For eight weeks the thing that plagued me most wasn’t a specific bug — it was the fact that fixing something didn’t fix it. Change one agent and the change would clash with another and contradict it, so the overall pipeline output didn’t move at all. Fix the Writer and the downstream caption step overwrote it (which is why a single-pass RF rewrite became meaningless within a day). Fix the generator and the reviewer (Giri) passed it by the old standard. Dig into one lane and the neglected other lane quietly regressed — until the next batch had “RF completely broken.” For a change to truly land, its ripple had to be reconciled across every stage of both lanes (Writer → Director → Camera → review). The end wasn’t “when I edited one file”; it was “when both lanes told the same story.” So I stopped leaving this to human discipline and encoded the change-impact trace as a skill (pipeline-change-impact) that forces any change to propagate to, verify against, and report on every consumer.

The second root was inside the prompt .md files themselves. Every time a PD review came in, I'd mindlessly append the note as "PD on such-and-such date: '…'" — and the dated band-aids piled up and contradicted each other. Yesterday's rule fought today's, nothing generalized, and I had to repeat the same corrections. The thicker the prompt got, the worse the pipeline behaved. The fix was to stop transcribing reviews and instead integrate them into a single principle → why → how → one example, superseding contradictions (git keeps the history), then bake that discipline into a skill (prompt-authoring) so the system governs the evolution of its own rules. The way this very retrospective is written — synthesizing review logs into principles rather than transcribing them — is a product of the same discipline.

Reliability and cost, engineered

Deterministic gates. When I stopped trusting the LLM reviewer, I stopped trying to fix it with prompt text and started computing signals in code and feeding it booleans — temporal grounding, “preachy caption” detection, era/existence coherence, character-marking checks. Prompt text alone won’t fix a reviewer. Compute the signal and feed it.

Failure-signature triage. Immediate ENOSPC = disk; a 600-second run = resource contention; an infinite block = a held lock. Just naming the signature turned days of debugging into minutes.

LLM calls are a cascade. Every LLM API call is wired as a fallback chain — if one provider blocks (e.g., Gemini returning an empty block) or times out, the call flows to the next model so the whole pipeline doesn't stall on a single point.

Cost ledger + caps. Every provider call is logged, retries are capped, and the expensive render step sits behind cheap validation. Across the codebase, fix commits (110) outnumbered feat (89) — an honest picture of what firefighting at the edge of a new stack looks like.

The spine — the failures I kept

The part I’m proudest of is the part most people delete. I keep a log of everything I rolled back, because deleted failures repeat. A few:

3D-modeling grandma’s living room (VLM → Blender). To lock background consistency, I fed a hand-drawn floor plan and 67 background photos into Gemini, extracted a 3D room layout, and even built it in Blender — then just switched to more photoreal real-photo references. Sophisticated modeling is not the shortcut to realism. Real source is.

The rubber-stamp reviewer — and its opposite, runaway re-renders. The LLM reviewer (Giri) was dangerous in two ways. One was rubber-stamping: it passed an obviously flawed batch 9/8/8/9 and auto-scheduled it. On sparse frames (two per cut), “praise” was the LLM’s default, and it even scored preachy captions as “compliant.” One day an entire day’s batch (4 videos) was scrapped. The other way was the exact opposite — when Giri ordered “revise / regenerate,” that one word was a ~$50 Seedance re-render, and a self-healing retry loop would re-render the same broken cut two, three times, with cost spiraling effectively to infinity. So the reviewer was either passing the wrong thing, or insisting it was right and burning money. Both faces pushed review wholesale onto deterministic gates (code computes the signal and feeds a boolean, not prompt text), and re-renders got a hard cap plus a “$0 caption-rewrite first” rule.

The model kept proposing “avoidance”; I kept pushing “raise the quality.” One pattern repeated: on hard problems (Ryani’s markings, background drift, photorealism), the AI collaborator would usually first suggest reducing scope or working around it (“let’s keep this cut static,” “let’s turn that feature off,” “let’s just compromise here”). Almost every time, I re-proposed it in the direction of raising quality. This is exactly why a human belongs in the loop: automation defaults to the easiest passing path, but the channel’s taste and standard have to be held by a person. That push-and-pull was the core role a human played across the eight weeks.

The hardest creative problem of the project — Ryani’s white markings. Getting my dog’s exact white markings right in every generated frame (chin, chest, paws; the nape pure black) without hallucination. A real pet channel never faces this — it films instead of generating. Strict gates killed good cuts; loose gates let bad markings ship. I barely found balance by lowering the pixel cap to advisory, comparing against reference images, and self-healing per cut. Markings alone account for 14+ commits — the single most-fought topic in the project.

The sync-and-disk-staging trap — the longest fight, and the one I never fully closed. My Mac had ~32GB free; the source archive was tens of gigabytes and growing. Once, a bulk export ignored its date range and pulled the entire set of pruned originals at once — 37GB — filling the disk to 100% (ENOSPC). The obvious fix, “watermark sync, only pull new items,” quietly created a worse problem: ~8,000 older clips were permanently excluded from ingest, so the Writer drew stories from a recency-biased pool and the oldest, most tender footage never surfaced (and because the iCloud shared album was never cleared after ingest, the sync kept re-seeing the same items and the cloud upload never fired). The real fix was a chunked backlog sync: pull ~10GB of un-ingested items, tag them, delete the staged copies, and repeat until the diff is empty — bounded by disk, not compute. That tagged the full library (~16,000 assets). But that very constraint — you can’t stage the whole archive at once — is why the structural fix, “move everything to the cloud,” still isn’t done: the migration is written and ready but unexecuted, blocked by the exact disk ceiling it’s meant to remove. The real ceiling of a local-first system wasn’t CPU. It was disk staging capacity.

Additive guardrails degrading the whole. A night’s worth of well-meant “improvements,” stacked, made the RF lane worse. New gates are kept only after they’re proven to help.

Launch batches stopped by payment failures. Because my payment method was a debit card, right around the late nights when I worked most (11:30pm–midnight), payment often failed — and the launch batch scheduled for exactly that hour didn’t run on many days. This isn’t memory, it’s on record: failures on the same card (ending 7043) clustered around 11:30pm–midnight (KST) across BytePlus, OpenAI, Anthropic, and Google, and once Anthropic even turned API access off on credit exhaustion. However robust the code, a pipeline’s real single point of failure can be one personal debit card. Automation only runs as far as the weakest piece of real-world infrastructure it stands on.

What’s left (the unfinished parts)

The studio runs, but it isn’t finished. The big open items:

• Cloud migration (GCP). The fundamental fix — lift the bots and batches off the Mac to run 24/7. The code is ready but blocked by the disk-staging ceiling above; it’s only complete once the one macOS dependency (iCloud ingest) is solved too.
• Kill the payment single point of failure. Move from a debit card to prepaid credits / a comfortable limit, so batches don’t stall on payment at night.
• The quality bottleneck has moved. With render stable, the bottleneck shifted back to direction and shot variety — I’m growing the Conte (cue-sheet) agent.
• Turn review into a 3-platform consensus harness. Today review is a single model (Giri) plus deterministic gates, but the single-model bias recurred — rubber-stamping defects, and conversely rejecting perfectly good photoreal output. Next is a review harness that combines the judgments of OpenAI, Anthropic, and Gemini: pass only when all three agree, so no single model’s whim can decide a batch.
• From shorts to long-form. Eventually leaving the 20-second vertical short for longer video — where cut continuity, narrative, and audio all have to hold far longer. It’s the next step for the whole pipeline.
• Audio via TTS. Today it’s burned-in captions + external BGM; I’ll move to text-to-speech narration so a narrator’s voice can lead the video.
• Seasonal bumper auto-update. Intro/outro bumpers are fixed today; they should auto-refresh by season (holiday, weather, trend), wiring the same timeliness signal as the arc season-plan into the branding assets.
• Small follow-ups. Strip the audio Seedance emits so it doesn’t collide with external BGM, reduce human+agent git friction, and fully close the bandit loop (measure → auto-schedule next month).

And the last line of this whole roadmap isn’t a technical item — it’s a person: handing operation to my parents. That’s in “The real goal” below.

What this actually shows

Strip away the pets, and this is an end-to-end multimodal AI production system: ambiguous inputs, model constraints, human review, deterministic gates, cost control, launch scheduling, and feedback loops — all bound into a single repeatable operating system. That’s the work I want to keep doing: taking an ambiguous, moving problem and building the structure that keeps it from falling apart. The studio is still running.

The real goal — my parents’ channel

Honestly, all this architecture is a means, not the end. What I really want to build is a channel that my parents, in their 70s, operate together with an AI agent.

And this isn’t someday-imagination — it’s already started. My parents upload the day’s footage and photos of the pets (Ryani and Leo) to the grandmompapa Slack channel, toss the bot episode ideas (“how about this today?”), and chat with it. Once, when the bot kept mis-drawing a prop (Leo’s toy — a dried-persimmon stem), they photographed the real thing and posted it to that channel themselves. And then the whole family watches the finished shorts together and laughs.

The technology recedes, and what’s left is everyday joy — talking about what to film today, reacting to a finished clip, the family chat lighting up more often over the dog and the cat. Relatives who live far away watch the same screen and laugh too. One channel becomes a small excuse to connect three generations, positive communication goes up inside it, and everyone is a little happier.

The most joyful moment of this project wasn’t a benchmark score. It was my parents genuinely delighting in seeing those exact two — Ryani and Leo, the pair they live with every day — inside little AI-made stories. They recognize, at a glance, tailless Ryani wiggling her rear and mischievous Leo getting into trouble on screen, laugh and say “oh my, that’s totally our babies,” replay it again and again, and send it to relatives to show off. That the real dog and cat they live with become the leads of a new little story tuned to each season and holiday — I never imagined my parents would love it this much. That small thrill of opening the channel in the morning to see “what are our kids up to today” was bigger than any metric I built. In the end, what I built over eight weeks wasn’t a video pipeline — it was a machine that automatically manufactures reasons for a family to enjoy something together.

My parents already produce the data, give the ideas, and talk with the bot. One day my mom, puzzled, asked, “Why does this thing read and reply so fast? I’m over here typing everything out by hand.” I told her it’s because of the chips made by Jensen Huang’s company — the ones on TV — that process a lot of things at once. “Technology really is amazing,” she said, delighted. Maybe that’s exactly how a senior becomes AI-native: not by understanding how it works, but by using it every day, marveling at it, and letting it into their life. So the last step left is clear too — put operation (review, scheduling) within their reach through the same Slack bot. Senior-friendly UX — an upload-a-photo-and-done flow, big type, short commands, low-friction review — is the final gate. Having my parents run the whole studio on Slack alone, even after I’ve stepped out of every seam — handing this studio from my hands to theirs — that’s the real finish line.

By the numbers: 8 weeks · 34 agent modules (files) · ~25,000 LOC · 343 commits / 27 days · 118 uploaded · 4 videos/day · pure-pipeline cost ~$2,700 (total provider charges $4,000+) · 12+ documented rollbacks · sora-2 → Veo → Seedance 2.0.

References & Notes

Data sources

• Dev metrics (commits, LOC, file counts): git history (repo baseline 2026–06–04 → 07–03). The May “web era” predates the repo, so its dates and figures are approximate.
• Cost: provider billing receipts (OpenAI, BytePlus, Google Cloud, Anthropic; received via Gmail) plus an internal cost ledger (api_calls table, precise from 6/25). Where the two disagree, I trusted the receipts.
• Performance: YouTube Analytics (48-hour views and retention) → video_performance.

Measurement caveats (honest limits)

• “34 agent modules” means 34 .py files under agents/, not 34 independent autonomous agents. LOC (~25,000) is a size reference, not a quality metric.
• “$4,000+” is total provider charges — it includes the Claude Code dev tooling that built the studio and other projects (biometrics, Firebase) on the same GCP account. The amount that attributes cleanly to the pipeline is ~$2,700 (Seedance + OpenAI), and that number is more defensible.
• GCP amounts are approximations converted at ₩1,380/$, and the pipeline vs. non-pipeline split is not exact.
• The “11:30pm–midnight” clustering of payment failures is an observation from receipt timestamps; the cause (card limits / reset time) is inferred.
• Commit-based metrics like fix > feat depend on message conventions and are directional only.
• 118 is an upload count; some were later made private or deleted via PD /veto.

Available on request: commit log, module map, architecture diagram, production and cost ledgers.