Building a Real-Time AI “Roast-Master” with Vision Agents & Python

TL;DR

• You are building a Python service that receives live webcam video and microphone audio, evaluates what is visible, and replies with spoken commentary while the user is still on screen.
• The hard problems (WebRTC negotiation, echo cancellation, jitter buffering, and turn detection) are handled by Vision Agents on top of Stream’s edge network, not by your code.
• All “behavior” lives in a system instruction. No classifiers, no fine-tuning, and no frame-by-frame logic beyond basic throttling.

Introduction

A “Roast-Master” is a real-time AI agent that watches a live camera feed and reacts out loud to what it sees. If your desk is cluttered, your lighting is harsh, or your camera angle is questionable, it comments immediately, while you are still on screen. There is no image upload and no delayed response. The entire interaction happens live.

This tutorial shows how to build that kind of agent using **Vision Agents and the [OpenAI Realtime API](https://developers.openai.com/api/docs/guides/realtime)**. Vision Agents handles WebRTC, audio cleanup, and interruption logic. OpenAI’s realtime model handles visual reasoning and spoken output. The Python code focuses only on behavior.

According to Cisco’s latest public Visual Networking Index, video accounts for over eighty percent of consumer internet traffic, with live and interactive formats growing faster than uploads. That distinction matters for developers because latency changes how systems are perceived. A delayed response feels like a batch tool and an immediate response feels reactive.

You can see this tension play out in Reddit communities such as r/MachineLearning. Builders regularly get image-based reactions working, then stall when they move to live video. Threads drift into WebRTC signaling, audio feedback loops, browser quirks, and timing edge cases. The “AI” part becomes the smallest piece of the system.

This guide takes a different approach i.e., you are not building a vision pipeline from scratch. You are defining constraints around attention, timing, and tone, then letting a realtime vision model operate inside those bounds. The Roast-Master is a concrete example, but the underlying pattern applies to any live, spoken video agent.

In the next section, we move from concept to behavior and describe exactly what this agent does on each observation loop and why those decisions matter.

What You’ll Build

You will build a live, interactive AI agent that watches a user through their webcam and reacts out loud to what it sees. The purpose of the project is not visual detection for its own sake, but to demonstrate how to build a low-latency, personality-driven video agent that responds in real time, without waiting for uploads or explicit prompts.

In this tutorial, a “Roast-Master” refers to an agent that continuously observes a video feed and delivers spoken commentary the moment it notices something visually obvious or awkward. That might be clutter on a desk, poor lighting, an unflattering camera angle, or an outfit choice that stands out. The agent does not wait for the user to ask a question and does not attempt to be helpful. Its behavior is reactive, interruptive, and intentionally opinionated.

From a technical standpoint, this Roast-Master is implemented as a single Python service. The service accepts a live WebRTC session from a browser, samples frames from the incoming video stream, sends those frames to a realtime vision language model, and streams generated speech back to the same session. The agent runs continuously for the duration of the call and reacts as visual context changes.

The key idea is that this project shows how to build an always-on video agent that feels present rather than transactional. The roast persona is simply an accessible way to make timing, interruption, and spoken output obvious during testing.

Roast Master demo Video

[embed]Stream_Roast_Bot_Demo_1.mp4 Edit descriptiondrive.google.com

Live Video Interaction

The user connects through a browser and shares their camera and microphone. From their perspective, the experience feels like a video call where the other side happens to be an AI agent. As they move, adjust their setup, or change rooms, the agent continues to observe and react without restarting the session.

Scene-Level Visual Reasoning

The agent does not track objects explicitly or maintain a list of detected items. Instead, it relies on a realtime vision language model to interpret each sampled frame as a complete scene. Judgments such as clutter, lighting quality, or visual imbalance are handled at the scene level, which is closer to how a human would assess the same environment.

Spoken Reactions as the Primary Output

All responses are delivered as audio. The agent does not generate text responses that are later converted to speech. When the model decides to comment, speech begins immediately and is streamed back to the user in the same realtime session. This keeps responses tightly coupled to what the user just did or revealed on camera.

Conversational Flow and Interruption

The agent shares the audio channel with the user. When the user speaks, the agent pauses its output. When the user stops, the agent is free to resume. This creates a natural back-and-forth without explicit turn prompts and allows the Roast-Master to interrupt when appropriate, then yield when the user responds.

By the end of this tutorial, you will have a working reference for building realtime, spoken video agents that react continuously to what they see. The roast persona makes the behavior obvious, but the same structure applies to reviewers, coaches, or any agent that needs to observe and speak in the moment.

Prerequisites

This project assumes a local development setup that can handle realtime audio and video.

Python and Package Manager

• Python 3.10 or newer
• uv (recommended) or pip

The Vision Agents SDK includes native media dependencies, so a clean virtual environment is required.

Required API Keys

You need credentials for two services:

You need credentials for two external services before running the agent.

• Stream API key and secret: These are used to establish and terminate WebRTC sessions at the edge. You can create them by signing in to the **GetStream** dashboard and creating a new application. The keys are available under the app’s API settings.
• OpenAI API key: This authorizes access to the realtime vision and audio model. You can generate an API key from the **OpenAI** dashboard once realtime access is enabled for your account.

Local Hardware

• A webcam and a microphone
• A modern browser with WebRTC support

No front-end build tools are required. The demo uses the Stream testing interface to establish a session with the local Python process.

Architecture Overview

Visual flow:

Media Ingress and Cleanup (Browser → Edge)

A browser sends camera and microphone tracks over WebRTC. You do not deal with codecs, packet loss, or signaling. That traffic is terminated at Stream’s edge, where echo cancellation and basic audio cleanup are applied before anything reaches your service.

From the Python side, the result is a steady stream of audio and video frames with timing already aligned. The application never sees browser quirks, device differences, or network jitter. That work is already done by the time data reaches your process.

Control Plane and Routing (Edge → Vision Agents)

Once media leaves the Stream edge and reaches your Python process, Vision Agents becomes the component that coordinates everything that happens next. It does not perform visual analysis itself. Instead, it controls the flow of audio and video through the system and decides when each part of the pipeline is allowed to act.

On the video side, Vision Agents determine which frames are sampled from the incoming stream and forwarded to the realtime model, based on the configured frame rate. Frames that are not sampled are discarded and never sent upstream. This keeps usage predictable and avoids unnecessary processing while the session is active.

On the audio side, the SDK continuously monitors incoming microphone audio to detect when the user is speaking. While user speech is present, audio output from the model is suppressed. When the channel is clear, generated speech is allowed to flow back to the browser. This gating happens automatically as part of the agent runtime.

Vision Agents also handle routing in both directions. It sends sampled frames to the realtime model, receives generated audio in response, and streams that audio back through the existing WebRTC session. Your application code does not manage buffers, playback timing, or synchronization. The agent object owns that loop end to end, allowing your code to focus only on configuration and behavior.

from vision_agents import Agent, StreamEdge
edge = StreamEdge()

The StreamEdge represents the boundary between the browser and your Python service. It terminates the WebRTC connection coming from the client and exposes a clean, server-side interface for receiving audio and video streams.

From the application’s point of view, the edge hides all browser and network complexity. Device differences, packet loss, echo cancellation, and timing alignment are handled before media reaches your code. What the agent receives is a normalized stream of audio and video that can be sampled, routed, and gated without dealing with WebRTC internals. This allows the rest of the service to treat live media as an input source, rather than as a networking problem.

Perception and Speech (Vision Agents → Realtime Model)

Visual analysis and speech generation are handled by the same realtime model session. Instead of sending images to one endpoint and converting text to audio in a separate step, the agent maintains a single, continuous connection where video frames go in and spoken responses come out. This keeps timing consistent and avoids extra latency that would otherwise be introduced by chaining multiple services.

from vision_agents.llms import OpenAI
llm = OpenAI.Realtime(
 model=”gpt-4o-realtime-preview”,
 fps=1
)

This configuration tells Vision Agents to use OpenAI’s realtime multimodal model as the backend for both vision and audio. The fps parameter controls how often video frames are sampled from the incoming stream and sent to the model. In this case, the agent forwards one frame per second.

For a Roast-Master, visual context changes slowly. Room layout, lighting, posture, and background clutter remain mostly stable across several seconds. Sampling more frequently increases usage without improving how responsive the agent feels. Setting fps=1 keeps the session efficient while still allowing the model to react immediately when something in the scene changes.

No Local Inference

There is no object detection model, vision pipeline, or neural network running on your machine. Frames are forwarded to the vision language model exactly as they are captured, and all visual interpretation happens remotely. Your Python service does not perform classification, tracking, or post-processing.

This matters because it removes an entire class of engineering work from the project. You do not need to manage model weights, GPU availability, or inference performance under load. There is no need to tune thresholds, maintain detection labels, or handle version drift between local models and deployed code.

It also keeps the system flexible because the agent is not tied to specific detectors, it can react to new visual patterns without code changes. Scene-level judgments such as “this setup looks cluttered” or “the lighting is harsh” are handled by the model’s general reasoning rather than by brittle rules.

From a deployment perspective, this keeps the service lightweight. The same Python process can run on a small CPU-only instance, since it is coordinating media and behavior rather than performing inference. That makes it easier to deploy, easier to scale, and easier to reason about when debugging realtime behavior.

Step 1: Environment Setup

This step prepares a local Python service to accept a live WebRTC session.

Initialize the Project

Create a new project directory and virtual environment using the ‘uv’ command.

uv init roast-bot
cd roast-bot
uv add vision-agents “vision-agents[openai]”

The [openai] extra installs the adapters required for the realtime model. Without it, the SDK cannot open a multimodal session.

If you prefer pip, the equivalent is:

pip install “vision-agents[openai]”

Configure Credentials

The service reads all credentials from environment variables at startup. You can provide them either by exporting variables in your shell or by loading them from a .env file during local development.

Option 1: Export environment variables

This approach works well for quick testing or when running the service directly from a terminal.

export STREAM_API_KEY=your_key_here
export STREAM_API_SECRET=your_secret_here
export OPENAI_API_KEY=your_openai_key_here

Once exported, these variables are available to the Python process for the duration of the shell session.

Option 2: Use a .env file

For local development, it is often easier to store credentials in a .env file that is loaded at startup.

Create a file named .env in the project root with the following contents:

STREAM_API_KEY=your_key_here
STREAM_API_SECRET=your_secret_here
OPENAI_API_KEY=your_openai_key_here

Make sure this file is not committed to version control. The application will read these values at runtime and use them to authenticate with Stream and OpenAI. These values are required before the agent starts. The Stream keys authenticate the WebRTC edge session. The OpenAI key authorizes the realtime vision and audio stream.

At this point, the environment is ready. The next step is to define the agent’s behaviour through system instructions and wire the agent object together.

Step 2: The Roast Logic (System Prompting)

In a realtime vision agent, the model is continuously receiving visual input. The system instruction does not control how fast the model processes frames, but it does control when the model decides to speak and what qualifies as worth reacting to. Tone is shaped entirely by the instruction.

When instructions are vague, the agent tends to hesitate. For example, an instruction like “comment on what you see” often results in long pauses or overly cautious descriptions because the model is unsure what counts as relevant. The agent may default to neutral scene summaries instead of reacting.

Overly verbose instructions cause a different failure mode. If the instruction lists too many rules, exceptions, or stylistic constraints, the agent may delay responses while trying to satisfy all of them. In practice, this shows up as late or rambling audio that feels disconnected from what just happened on camera.

Effective instructions narrow the agent’s scope. They clearly state what kinds of visual details deserve a response, what tone to use, and when silence is acceptable. This allows the agent to react quickly and consistently without second-guessing its role.

Why System Instructions Replace Traditional Logic

In a classic vision pipeline, you would detect objects, map them to rules, and trigger responses. That fails in conversational settings because context matters more than labels. A messy desk, harsh lighting, and an awkward camera angle are scene-level judgments.

Here are a few examples of them:

The realtime model already understands scenes. Your job is to constrain its behavior so it reacts in a consistent way.

Defining the Persona and Reaction Rules

The instructions below do three important things:

• It fixes the persona so the tone does not drift.
• It limits scope to visible details only.
• It forces immediacy, which prevents delayed commentary.

ROAST_INSTRUCTIONS = “””
Role:
You are a ruthless, high-fashion interior designer with no social filter.

Context:
You are continuously watching a live video feed of a user and their surroundings.

Objective:
Your only task is to roast what you see. Focus on visual mistakes, poor taste, and awkward setups.
Behavior rules:
- Comment only on things that are visible in the video.
- Prioritize obvious flaws: messy cables, bad lighting, cluttered desks, cheap furniture, awkward camera angles, or questionable clothing.
- Assume confidence. Never hedge or soften a comment.
- Speak in short, punchy sentences meant to be heard, not read.
- Deliver observations as statements, not questions.
- Do not explain yourself.
- Do not offer advice or fixes.
- Do not be friendly, polite, or encouraging.

Timing:
- Speak immediately when something roast-worthy appears.
- If nothing stands out, stay silent.
Output style:
- Sound sarcastic, judgmental, and dismissive.
- One observation at a time.
- No greetings. No closings. No filler.
“””

There are no safety disclaimers, no conversational openers, and no confirmation requests. This keeps latency low and speech direct.

When working on similar agents, most iteration happens here. Small wording changes can affect timing more than any code change.

Step 3: Initializing the Agent

With the instruction defined, the agent wiring is straightforward. The Vision Agents SDK connects the media edge, the instruction, and the realtime model into a single loop.

Agent Construction

from vision_agents import Agent, StreamEdge
from vision_agents.llms import OpenAI

agent = Agent(
 edge=StreamEdge(),
 instructions=ROAST_INSTRUCTIONS,
 llm=OpenAI.Realtime(
 model=”gpt-4o-realtime-preview”,
 fps=1
 )
)

This is the entire runtime definition. There is no separate vision client, no audio player, and no session manager in your code.

Why fps=1 Is Enough

Frame rate controls how often the agent samples the video stream. Higher values increase cost and context churn without improving perceived awareness for this use case.

For tasks such as sports analysis or navigation, frame-to-frame motion is crucial. For evaluating a room, posture, or lighting, it does not. One frame per second is enough to maintain the illusion of continuous sight while keeping usage predictable.

This parameter is one of the main cost controls in realtime vision agents.

Step 4: Handling Turn-Taking

A roast bot that constantly interrupts the user feels broken. One that never resumes feels passive. Turn control is what keeps the interaction usable.

The Interruption Problem

If the user starts talking, even briefly, the agent needs to stop speaking. Otherwise, audio overlaps and the session becomes unintelligible. At the same time, the agent should not permanently yield the floor.

Built-In Voice Activity Detection

Vision Agents includes voice activity detection that monitors incoming audio and gates output accordingly. You do not implement this logic yourself.

A typical configuration looks like this:

agent.configure_audio(
 interrupt_on_speech=True,
 resume_after_silence=True
)

With this feature enabled, the agent pauses when speech is detected and resumes once the channel is cleared. This allows behavior like brief defences followed by another comment, without manual timing logic.

Step 5: Running the Demo

Once the agent is defined, running the service is a single command.

python roast_bot.py

When the process starts, it prints a local URL for the Stream testing interface.

Connecting the Browser

• Open the provided URL in a browser.
• Grant camera and microphone permissions.
• The WebRTC session connects automatically.

There is no frontend build step and no client-side logic to maintain. The Stream debugger UI is sufficient for development and recording demos.

At this point, the agent begins observing and responding as soon as it detects something worth commenting on.

Conclusion

You now have a working reference for a realtime vision agent that reacts while a user remains on camera. The code footprint is small, yet it covers problems that usually require signaling servers, audio pipelines, and browser-specific handling. Stream’s edge network takes care of video ingress and audio timing. The OpenAI Realtime model produces vision and speech. The Python service only defines what the agent is allowed to do.

The roast persona is just one example. The real value sits in the structure. One system instruction controls tone, timing, and scope. One agent instance ties media, scene understanding, and spoken output into a single loop. That same structure works for any other persona without changing how the system is wired.

Next Steps

Replace the roast instructions with another role, such as a yoga coach watching posture or a design reviewer checking a workspace. The rest of the code stays the same.

Run the Python service on a small cloud host such as Railway or Fly.io so other people can connect to it through the same WebRTC flow.

FAQs

1. Does this record the user?

No. Video frames and audio are processed transiently as part of a live session. The agent does not store media or write frames to disk.

2. How much does it cost to run?

Vision Agents itself is free to use. Costs come from the OpenAI Realtime API, which is billed based on audio and token usage. Sampling video at a low frame rate, such as one frame per second, is the primary way to control usage.

3. Can I use a local model instead of OpenAI?

Yes. Vision Agents supports multiple model backends. That said, expressive spoken reactions depend heavily on strong multimodal reasoning, which current local models often struggle with.

4. Why use the Stream edge at all?

Latency determines whether spoken reactions feel natural or late. Stream’s edge network maintains stable audio and video timing across networks, which is crucial when responses are intended to appear immediately after an action is taken.