Avi — A Digital Twin Platform for Personalized AI Avatars

This article was produced as part of the final project for Harvard’s AC215 Fall 2025 course. You can access our GitHub or watch a video walkthrough here.

Background and Motivation

The information age has brought with it an abundance of ways to collect data that can be used to unlock valuable insights. Unfortunately, this data is often siloed in various different apps and websites, preventing individuals from being able to harness its full potential. Avi is a digital twin platform that helps individuals make their data work for them and experts scale their impact without losing their personal touch. For individuals, Avi connects data across services to unlock deeper insights, smarter recommendations, and more personalized experiences. For coaches, instructors, and creators, Avi’s intelligent digital twins handle the repetitive, foundational work, so they can focus on the last mile: the uniquely human touch that makes each client experience exceptional.

Avi connects data that is usually siloed across different platforms

Technical Implementation

Avi is implemented as a cloud-native digital-twin platform designed to integrate user data, generate personalized insights, and support real-time multimodal interaction with human-like avatars. Its architecture is organized into three coordinated layers: Process, Execution, and State.

Avi Solution Architecture

Process Layer — Human Workflows + ML Development: This layer captures how users and experts interact with the system, and how our team builds and improves it. Users onboard through a voice-guided setup flow where they (a) authenticate with Google OAuth, (b) connect optional data sources, (c) record sample audio for ElevenLabs voice cloning, and (d) customize a 3D avatar using Ready Player Me. Expert avatars are created from instructor transcripts and structured data provided through a professional onboarding tool. Behind the scenes, the team fine-tunes models that power conversation and expert likeness.

Execution Layer — Microservices + AI Services: All operational logic runs in backend microservices deployed on Google Kubernetes Engine (GKE). A lightweight API layer enables browsers and mobile clients to communicate with the backend without directly handling model calls. These services support:

1. Conversational engines using Gemini 2.5 Flash hosted on Vertex AI.
2. Supervised fine-tuning (SFT) of Gemini 2.5 Flash through the Vertex AI Tuning API generates expert models aligned with authentic coaching patterns and instructional tone.
3. Data-processing pipelines for ingestion, normalization, embedding generation, and retrieval-augmented generation (RAG).
4. Voice cloning and playback through ElevenLabs APIs.

State Layer — Persistent Context + Personalization: To preserve shared context between individuals and experts, Avi stores user data across multiple specialized storage systems. Cloud SQL maintains structured identity and metadata; a vector database stores embeddings for RAG-based personalization; and Google Cloud Storage holds media such as audio clips, avatar models, and generated assets. Persistent source control and a container registry version of the system itself, ensuring reproducible deployments and traceability for model updates.

Together, these layers create a modular, scalable architecture where digital twins continuously learn from personal behavior.

Avi Technical Architecture

Application Interface

Avi’s interface is built around natural, human-like interaction with digital twins. Users can talk to their avatar through voice chat, ask questions, track progress, or reflect on goals in a conversational format. A traditional text-based chat option supports quieter or multitasking use, while a friends feature enables users to add others and interact with their digital twins socially. A dedicated expert-avatar area provides voice-based coaching from a real instructor’s digital twin, offering personalized guidance and answering questions in real time. For fitness use cases, the expert can also generate individualized training plans that adapt as the user progresses, creating an interactive, data-driven coaching experience.

Interacting with friends

Expert fitness instructor avatar

Technical Efficiency

Avi is designed for scalable efficiency as both users and data sources grow. Its containerized architecture on Google Cloud Platform autoscales services such as avatar generation, voice processing, and model inference, minimizing compute waste. The platform expands horizontally with demand, integrating new data sources when they meaningfully enhance guidance. As user volume increases, batch ingestion, embedding updates, and insight generation will shift to more frequent scheduled jobs. This approach allows Avi to scale responsibly while maintaining high-quality, personalized experiences.

Future Development

Avi’s development roadmap focuses on expanding both the depth and reach of digital twins. In the near term, the platform will grow into an ecosystem of expert avatars across fitness, education, and creative instruction, paired with experiments in pricing models and partnerships with gyms, studios, and creator businesses. Long-term development moves toward enriching the social layer where users interact not only with their own twin, but with the twins of friends and professionals. Over time, this infrastructure can support applications beyond coaching — such as dating, language learning, or personal finance — ultimately becoming a plug-and-play platform for personal data-driven experiences.