I Built an AI That Can Actually Watch Over You. Here’s the Full Story of How OmniAi Came to Life

This article was created for the purposes of entering the Google Gemini API Developer Hackathon. #GeminiLiveAgentChallenge

— -

There is a moment in every developer’s life when a project stops being about the code and starts being about something bigger. Something that genuinely matters. For me, that moment came somewhere around 2 a.m., watching my phone’s AI agent spiral into an infinite loop, talking to itself, responding to its own voice, getting increasingly confused, until the whole thing collapsed. I sat there for a moment, then started laughing.

Because as broken as it was, I knew something right then: I was building something that actually lived and breathed. Something that wasn’t just generating text. It was perceiving the world.

That’s the story of OmniAi.

— -

THE ITCH I COULDN’T SCRATCH

I’ve been building software for years, and I’ve used almost every AI tool you can name. But every single one of them shares the same fundamental UX pattern: you type something, you wait, you get a response. Even voice assistants follow this rhythm. You speak a command, there’s a pause, and then you get an answer.

That pause has always bothered me.

Not because it’s slow, though it often is, but because it makes AI feel like a vending machine. You insert a coin, you get a result. There’s no sense of presence. No sense that the thing is actually with you.

I started wondering: what would an AI that was genuinely present look like? Not one that waits for input, but one that’s already watching. Already listening. Ready in the same way a trusted person sitting next to you is ready, not hovering, not intrusive, just… there.

I thought about use cases that made me feel something:

A grandmother trying to read the tiny text on the back of a medication bottle at night, wondering if this pill is the right one.

A young person walking alone late at night in an unfamiliar part of the city and genuinely wishing someone dependable was watching over them.

Someone sitting across the table from a used car salesman, a lease agreement in front of them, not sure if the clause they’re reading means what they think it means.

A remote worker staring at a screen for the sixth hour, drifting to Twitter for the fourth time, needing something gentle to pull them back.

None of these people want to “open a chat interface.” They just need someone to be there. Someone who sees what they see and can quietly say, “Hey, look at this.”

That’s why I built OmniAi.

— -

WHY THE GEMINI MULTIMODAL LIVE API IS DIFFERENT

Before I talk about what I built, I have to explain the technology that made it possible, because without it, the whole vision would have been impossible.

Most AI voice systems are daisy chains. Your voice gets transcribed to text by a speech-to-text model. That text gets sent to a large language model. The LLM generates a text response. That text gets synthesised back into speech by a text-to-speech model. Then it plays back to you.

Each link in that chain adds latency. Each conversion step also loses information, tone, emotion, emphasis, hesitation, things that are linguistically meaningful but get destroyed when you convert audio to text.

The Gemini 2.5 Flash Native Audio model breaks that chain entirely. It takes raw audio in, reasons directly over it (and over images, video frames, and text simultaneously), and outputs raw audio. One model. One pass. The result is a conversation that feels genuinely responsive in a way that transcription-based systems simply cannot replicate.

And beyond speed, there’s understanding. When you mumble, when your voice rises with surprise, when you laugh while you’re asking a question, the Native Audio model hears all of it. It doesn’t strip out the signal to extract the words. It processes the signal itself.

The moment I saw a demo of what Native Audio could do in a real-time visual grounding scenario, where the model was simultaneously watching a camera feed and holding a spoken conversation, I knew this was the foundation I’d been waiting for.

I applied for access to the Gemini Live API, got it, and immediately started hacking.

— -

THE GOOGLE AGENT DEVELOPMENT KIT: THE PART EVERYONE SHOULD KNOW ABOUT

If you’re building anything serious on top of Gemini’s Live API, do yourself a favor and use the Google Agent Development Kit (ADK).

Managing a persistent, real-time Gemini session is not trivial. Sessions have a lifecycle. They need to be opened, sustained, reconnected on failure, and cleanly torn down. Audio and image data needs to be fed in through a queue , the LiveRequestQueue, and the responses need to be routed back to the user in the right order, without duplication, without gaps.

I initially tried to manage all of this manually. I got a working prototype in about two days, and then spent another five days chasing race conditions, duplicate responses, and mysterious session-drop bugs.

Then I switched to ADK. The Runner, ADK’s session management abstraction, handles reconnects automatically. The LiveRequestQueue handles backpressure. A lot of the plumbing that had been killing me just… disappeared. The framework was clearly designed by people who had already made all the mistakes I was making.

ADK also made tool dispatch clean. In OmniAi, the agent has a full toolkit it can invoke: remember (saves user preference to the database), open_webview (opens an in-app browser), webview_tap, webview_scroll, webview_type, webview_screenshot, generate_image, generate_video, and others. ADK routes tool calls to the correct Python functions and injects the results back into the model’s context automatically.

I went from wrestling with infrastructure to actually building features. That shift is everything.

— -

THE ARCHITECTURE: EXPLAINED SIMPLY

Before I get into the chaos of what was hard to build, let me describe the system structure at a human level.

OmniAi has three layers:

Layer 1: The Flutter Mobile App

This is what the user actually holds. It runs on both iOS and Android and does several things simultaneously:

• It continuously captures camera frames (JPEGs) and streams them to the backend over a WebSocket

• It captures microphone audio as raw PCM at 16kHz and streams that too

• It plays the agent’s audio responses through the device speaker

• It runs an in-app browser that the agent can take control of

• It does on-device speaker diarization to detect when the user is speaking versus when the AI is playing back

All of this happens in parallel, orchestrated through GetX state management and a dedicated Socket.IO client.

Layer 2: The Python Backend

This is the spine of the system. Flask handles incoming Socket.IO connections. Each connected user session spawns its own dedicated asyncio event loop, running in its own thread.

Within that loop, the backend takes the incoming audio and camera frames and feeds them into Gemini’s Live API via ADK’s LiveRequestQueue. The model’s audio and tool-call responses come back through ADK’s runner and get routed back to the right client.

The backend also hosts the business logic for all the agent’s tools. When the agent decides to save something to memory, a Python function writes it to MySQL. When it decides to tap a button in the webview, a command fires to the Flutter client, which performs the action and sends back a screenshot so the agent can see what changed.

Layer 3: Google Cloud

The entire backend runs on Google Cloud Platform, behind Nginx acting as a reverse proxy and Gunicorn serving the Flask application with threaded workers. Cloud Storage holds generated media assets. Vertex AI is called when the Proactive Storyteller mode runs, either Imagen for generating still images or Veo for generating short AI videos. These get uploaded to Cloud Storage, and their URLs are sent back to the Flutter app for display.

The whole stack is real, deployed, and accessible from a physical phone.

— -

THE TWELVE MODES: ONE AGENT, MANY EYES

One of the design decisions I’m most proud of is that OmniAi doesn’t have twelve separate agents. It has one agent with twelve operational contexts.

Each mode is essentially a “lens”, a different system prompt preamble, a different set of active tools, a different conversational posture. The underlying Gemini model is the same. The session is continuous. When you switch modes, the agent shifts its frame of reference and continues the conversation.

Here’s what each mode does:

1. Surroundings Narrator; For visually impaired users. The agent watches through the camera and continuously narrates what it sees. Objects, people, text, hazards. No prompt required once activated.

2. Pill Identifier; Point your camera at a medication. The agent reads the imprint, identifies it, looks up its purpose, and tells you whether there are known interactions or warnings to be aware of. It does this by combining visual grounding with its medical knowledge base.

3. Bill Auditor; Hold your receipt up to the camera. The agent reads every line item, adds up the math, checks for service charges that weren’t disclosed, and reports back. It’s surprisingly satisfying to use.

4. Scam Shield; A passive monitoring mode that listens to phone call audio or watches a shared screen for language patterns and visual cues associated with scams — urgency manipulation, threats, requests for gift cards, suspicious countdown timers, etc. When something looks coercive, it quietly alerts you.

5. Contract Reader; Feed it a document by sharing your screen or pointing the camera at printed text. It identifies clauses that are unusual, explains what the legalese actually means, and flags anything that gives you less protection than you’d typically expect.

6. Form Coach; Watch yourself work out. The agent observes your posture, counts reps, identifies form issues (“your left elbow is dropping on the press”), and coaches you through the set in real time.

7. Price Hunter; Tell it an item you want to buy. It opens the in-app browser, searches multiple retailers, compares prices, checks reviews, and reports back. It can navigate purchase flows autonomously to check shipping and final cost.

8. Booking Concierge; Give it a date, a city, and a preference. It accesses booking sites through the internal browser, navigates to availability calendars, checks options, and reads back what it finds. No copy-pasting. No switching apps.

9. Situational Guardian; Silent mode. The agent monitors your environment passively and only speaks if it detects something worth flagging, someone approaching behind you, a sign that contradicts what you were told, a facial expression of distress. Designed for situations where you can’t actively interact with your phone.

10. Proactive Storyteller; After a session of any kind, the agent reflects on what it observed, the places you went, the things it saw, the topics you discussed, and generates an illustrated narrative. It composes a story, commissions images from Imagen, and assembles a small visual report you can keep or share.

11. Focus Anchor; Screen monitoring mode. The agent watches what’s on your screen using shared screen capture. If you’re supposed to be writing and you open Reddit, it gently says something. It knows what you’re working on because you told it. It’s less a surveillance tool and more like having an accountability partner.

12. Caregiver Eye; Safety monitoring. Designed for elderly care or child safety contexts. The agent passively watches a room and listens for sounds that might indicate a fall, a lack of movement for an unusual period, or signs of distress. It can be configured to alert a caregiver by sending a notification.

— -

THE CHALLENGES: THE PARTS THAT ACTUALLY HURT

Building OmniAi was not a smooth arc of progress. Here are the battles that genuinely consumed me.

— — The Echo Spiral — -

On the very first real device test, something went wrong almost immediately. OmniAi started speaking through the phone speaker. The microphone picked up that audio. Gemini heard its own voice coming back as input. It started responding to itself. The responses got longer. The loop tightened. Within about fifteen seconds, the agent was having a full conversation with itself in an increasingly disoriented state.

I stopped it, stared at the screen, and said out loud to no one: “I’ve created a confused AI.”

The fix was a technique I ended up calling PCM Ducking. While audio is being played back through the speaker, I drop the microphone input gain to 0.2, essentially making the speaker audio nearly inaudible to the mic. Combined with enabling hardware echo cancellation at the iOS AVAudioSession level (using .videoChat mode, which activates the AEC hardware path), the loop was broken.

But getting there involved understanding audio session categories, signal routing on iOS hardware, and the specific ways AVFoundation handles simultaneous capture and playback. None of it is well-documented for this exact use case. It took days.

— — The Audio Engine Dying Mid-Conversation — -

The next crisis was the audio playback engine falling over when the interaction tempo was fast. If the user asked a quick question and the agent gave a short reply, and then the user immediately asked another question, sometimes the playback would just stop. Not error, stop. Silence.

This turned out to be a race condition between concurrent operations on the audio player. When you call play() and stop() in rapid succession on some configurations, the underlying hardware engine can enter a bad state.

I built two solutions:

An Async Modification Queue, every operation on the player (play, stop, pause, seek) is serialized through a queue rather than called directly. This means concurrent calls can’t collide at the hardware level.

A Pulse Watchdog, a background timer that fires every several seconds and, if the audio engine has gone quiet and shouldn’t have, plays a brief completely-silent audio buffer. This “pulse” keeps the engine in an initialized state and prevents it from being cleaned up by the OS while it’s still needed.

It’s one of those solutions that feels a bit absurd when you describe it, but that works absolutely reliably in practice.

— — The Interruption Problem (Barge-In) — -

This was the deepest technical problem I solved in the entire project, and the one I’m most proud of.

Natural conversation requires the ability to interrupt. If someone is mid-sentence and you need to say something, you just talk. You don’t wait for them to finish. Any AI companion that doesn’t support this will feel clinical and robotic, not like a presence.

But implementing true barge-in is genuinely hard for several reasons:

First, how do you detect that the user has spoken and not the AI? When the speaker is on and the AI is mid-sentence, the microphone is picking up the AI’s own audio as well as any ambient sound. If you just do a volume threshold check, you’ll constantly false-positive on the AI’s voice.

I integrated sherpa_onnx, an on-device speaker diarization engine, into the Flutter app. It runs entirely locally, with no network round-trip, and can distinguish between different speaker identities in a mixed audio stream. When it identifies that the current audio is coming from the user’s voice (not the AI’s playback), it triggers the barge-in path.

Second, how do you actually stop Gemini from generating? This took me an embarrassingly long time to figure out. You cannot send Gemini a “stop” message. There is no cancel signal in the Live API client that reliably halts the output stream instantly. The only approach that worked was closing the LiveRequestQueue entirely on the backend. This terminates the session from the model’s perspective. The ADK runner then automatically reconnects for the next message. Clean break.

Third, there’s the stale audio problem. By the time barge-in is detected, the backend may have already sent several seconds of audio down to the client that’s sitting in a playback buffer. If you don’t clear that buffer, the user will interrupt the AI, the AI will stop generating, but the phone will continue playing the last few seconds of buffered audio as if nothing happened.

The fix: the moment barge-in fires on the client, the local playback buffer is discarded completely. Any audio chunks that arrive from the backend after the stop signal are silently dropped until the new session is established.

Three systems, on-device speaker AI, backend stream lifecycle, and client buffer management, had to coordinate in a window of tens of milliseconds. Once it worked cleanly, it felt like magic. You just talk. The AI stops. It listens.

— — Screen Sharing Across Platforms — -

Getting reliable, high-framerate screen capture working on both iOS and Android required writing native platform channel code for each OS.

On Android, the Media Projection API is the relevant system, but threading it through Flutter and making sure the captured frames are compressed efficiently enough to stream at useful framerates without overwhelming the WebSocket took a lot of tuning.

On iOS, the relevant mechanism is ReplayKit’s broadcast extension, which runs in a separate process from the main app. Getting frames from that process into the live agent session required inter-process communication through an App Group container.

Neither API is simple. Neither has a lot of Flutter-specific documentation. But once it worked, it enabled an entirely new category of assistance: the agent can now watch what you’re doing on your screen and help in real time, whether you’re filling out a form, reading an article, shopping, or coding.

— -

THE TECHNICAL STACK IN DETAIL

For completeness, here is exactly what I used and what each piece does:

Backend:

• Python 3.11+; core runtime

• Flask + Flask-SocketIO; HTTP API and real-time WebSocket server

• Google ADK (google-adk); agent orchestration, session lifecycle, tool dispatch

• google-genai; Gemini multimodal types and API client

• Gemini 2.5 Flash Native Audio; the core AI model

• SQLAlchemy + MySQL (via PyMySQL); persistent user memory and session storage

• Flask-Migrate; database schema migrations

• Google Cloud Storage; media asset storage

• Vertex AI Imagen; AI image generation in Proactive Storyteller mode

• Vertex AI Veo; AI video generation in Proactive Storyteller mode

• Nginx + Gunicorn; production server configuration on Google Cloud Platform

Mobile (Flutter):

• Flutter / Dart; cross-platform mobile framework

• GetX; state management and in-app routing

• camera; live camera frame capture and JPEG encoding

• record; microphone PCM audio capture at 16kHz

• just_audio + audio_session; low-latency audio playback and AVAudioSession configuration

• socket_io_client; WebSocket communication with the backend

• flutter_inappwebview; in-app browser with screenshot injection for agent control

• sherpa_onnx; on-device speaker diarization for barge-in detection

• Firebase; push notifications and analytics

— -

WHAT SURPRISED ME

A few things I didn’t expect:

The model is genuinely curious. When I give it a rich camera feed with interesting content, it doesn’t just answer questions, it proactively notices things. Without being prompted, it will say something like, “I noticed the price on that tag doesn’t match what the sign above it says.” That proactivity is what made me most confident this approach is the right one.

Memory matters more than I expected. Once I built persistent memory, where the agent actually stores and recalls facts about you, the experience transformed. There’s a psychological shift that happens when an AI remembers something you told it last week. It goes from being a utility to feeling like a relationship.

The audio quality issues were more engineering than AI. Almost all the hardest problems I faced had nothing to do with the model itself. They were about audio hardware, threading, race conditions, buffer management, and OS-level quirks. If you want to build real-time voice AI for mobile, prepare to spend serious time in the audio stack.

— -

WHAT’S NEXT

OmniAi is a working system, but it’s also a foundation. Here’s what I’m planning to build next:

Local RAG Integration, the ability to ground the agent’s responses in your own documents. Think of it as giving OmniAi a personal library that’s specific to you, your medical records, your lease, your tax documents, so it can give truly personalized guidance.

Wearable Hardware, the phone is a great first body, but it’s not the right final form. Smart glasses with a forward-facing camera and an earpiece would make OmniAi genuinely ambient in a way the phone can’t quite achieve.

Proactive Emergency Response, currently the agent monitors and alerts. The next step is giving it the ability to act. In genuine life-safety situations, a detected fall with no subsequent movement, a medical alert keyword in audio, the agent should be able to initiate an emergency contact call autonomously, without waiting for the user to respond.

Multi-user Memory Graphs, right now each user has a personal memory store. The next step is letting trusted users share context, so a caregiver can see what the care recipient’s agent has observed, or a family can maintain a shared situational awareness.

— -

A NOTE ON BUILDING WITH GOOGLE AI

I want to be honest about what it was like to build with this stack, rather than just listing the technologies.

The Gemini Multimodal Live API is genuinely new territory. The documentation is helpful but necessarily limited, this is a rapidly evolving surface. You will hit edge cases that no Stack Overflow answer covers. You will find behaviors that are only explainable by testing empirically. You will need to read SDK source code.

That said: the foundations are solid. The model is remarkable. The Native Audio capability, in particular, is not an incremental improvement on existing voice AI, it is qualitatively different. Conversation with it feels different, and that difference matters when you’re trying to build something that feels like a companion rather than a tool.

The ADK was the biggest positive surprise. I expected a thin wrapper. What I found was a thoughtful framework that had clearly absorbed real lessons from production deployments of conversational agents. Use it.

Vertex AI, specifically Imagen and Veo , integrated more smoothly than I expected for a first-time user. The Python SDK is well-structured, authentication flows cleanly if you’re already in the GCP ecosystem, and the models are fast enough to use in near-real-time content generation contexts.

Overall: this is an excellent stack for building serious multimodal AI applications. It’s not without rough edges, but what you get in return for navigating those edges is access to model capabilities that are, at this moment, genuinely world-class.

— -

If you build something with the Gemini Live API, I’d genuinely love to hear about it. The space is wide open right now. The technology is there. The interesting work is figuring out what to build on top of it.

OmniAi was my answer to that question. Your answer will be different. Build it.

— -

This article was created for the purposes of entering the Google Gemini API Developer Hackathon. #GeminiLiveAgentChallenge