Chaos Fit: Building a Real-Time AI Workout Coach with Gemini Live

How I used Google ADK, Gemini Live, and Google Cloud to solve the “bad rep” problem with bidirectional streaming.

I built Chaos Fit to answer one simple question: What if your workout coach could see your movement, hear you in real time, and interrupt with form corrections before those bad reps stack up?

Traditional fitness apps are passive; they play a video and hope for the best. When you’re a parent or just a busy professional, you don’t have time for a bad workout. Chaos Fit is active. It is a live coaching ecosystem that streams text, microphone audio, and webcam frames to an AI backend that can talk back, correct you, and even handle interruptions like a human trainer would.

This project and blog post were created specifically for the Google #GeminiLiveAgentChallenge hackathon submission.

The Google Cloud Stack

The Vision: A Duplex Coaching Experience

The key interaction pattern here is duplex. In a standard app, you click “Next.” In Chaos Fit, you speak naturally. The model responds in-flow with concise cues, supporting:

Live Webcam Streaming: Constant visual context for form feedback.

Interruption-Awareness: The coach can stop you mid-sentence if your back is rounding.

Persistent Sessions: Pause, resume, or end sessions without losing state or reconnecting. Coming soon!

How I Built It: The Google AI + Cloud Stack

1. The Real-Time Engine (FastAPI + WebSockets)

The backbone of the app is a bidirectional WebSocket endpoint (/ws/{user_id}/{session_id}). Inbound media events are normalized and routed through a LiveRequestQueue, while downstream model events are streamed back to the browser. This ensures the UI and the AI stay perfectly synced.

2. Bidirectional Orchestration with Google ADK

I utilized the Google ADK (Agent Development Kit) live runtime (Runner.run_live) to manage complex turn-taking. This was a game-changer for:

• Real-time streaming mode configuration
• Handling interruption events gracefully
• Maintaining session continuity

3. Gemini Live: The Brains and the Voice

Chaos Fit leverages Gemini Live models with a clever fallback system. The app can toggle between the Gemini Live API (via AI Studio) for rapid prototyping and the Vertex AI Live API for robust, cloud-native deployment. The native audio models enable true duplex conversation with natural interruption patterns.

4. Vertex AI: Optional Cloud-Native Fallback

For production deployments, the app supports Vertex AI as an optional backend when GOOGLE_GENAI_USE_VERTEXAI=TRUE. This provides cloud-native model serving for workout block generation and adaptive scheduling. The

SessionManager can use genai.Client(vertexai=True) to generate structured workout plans with JSON output, providing deterministic fallback logic when needed.

5. Cloud Firestore: Session Persistence

This is the core data persistence layer for ChaosFit. Every workout session is automatically saved to the session_summaries collection in Cloud Firestore, capturing:

• Exercise tracking: Current exercise type, rep counts, and exercise history
• Form corrections: All coaching interventions are logged for analysis
• Session metadata: Start/end times, interruption counts, and session goals
• Real-time events: All session state changes are stored in the events subcollection

The Firestore integration ensures that even if a session disconnects unexpectedly, all workout data is preserved and can be retrieved via the /reports/session/{session_id} endpoint.

Hard-Won Lessons from the Trenches

During development, a few “bugs” actually turned into features:

1 FPS is a Sweet Spot: I found that streaming frames at ~1 FPS produced significantly better latency and less browser backpressure than high-frequency video, while still providing ample context for form feedback.

Coaching ≠ Pose Estimation: Live coaching is about context. While ADK is great for feedback, I learned that high-resolution motion tracking is best served by a separate CV pipeline working in tandem with the AI’s verbal cues.

Life Happens (The “Pause” Button): In a home workout, interruptions are the norm. Building robust pause/resume controls wasn’t an edge case — it was a core product requirement. This was in version 1, but will bring back very soon.

Firestore Race Conditions: Multiple cleanup handlers can overwrite session summaries. I learned to always check session.status != "ended" before saving to prevent overwriting properly ended sessions.

What’s Next for Chaos Fit?

The foundation is set, but the ceiling is high. My next steps include:

Fixing a few buggys in the app

Automated Regression Testing: Specifically around interruption behavior.

Session Analytics: Tracking “correction acceptance” to see if users actually listen to the AI!

Advanced Scoring: Integrating a dedicated motion-analysis pipeline for granular form grading.

Enhanced Firestore Analytics: Building dashboards to visualize workout trends and coaching effectiveness across the stored session data.

Getting Started with Google ADK

I built ChaosFit starting from Google’s official ADK bidirectional demo as the foundation. The Google ADK bidi-demo provided the essential WebSocket patterns and the LiveRequestQueue implementation, which I then extended for fitness coaching.

From that starting point, I added:

• Firestore session persistence for workout tracking
• Real-time video frame streaming for visual context
• Session lifecycle management
• Exercise data extraction and summary generation

The bidi-demo’s clean separation of upstream/downstream WebSocket handling made it the perfect scaffold for building a real-time coaching application on top of Google’s ADK framework.

Here’s a link to the chaosfit repo: https://github.com/ElishebaW/chaosfit

Built for the Google #GeminiLiveAgentChallenge — demonstrating the power of Google’s integrated AI and Cloud platform for real-time applications.