Language Learning with Realistic Conversations

What would a voice-first language learning app built around real-world conversations look like?

The Language-Learning Problem

I spent hundreds of hours on Duolingo learning Japanese, driven by constant harassment from that little green owl. My streak went up. However, my confidence in Japanese did not. When it came to real conversations, I still struggled with basic tasks like asking for directions.

Joining the “1000 day learning streak” club on Duolingo — but at what cost?

Popular language learning apps like Duolingo often feel like a glorified flashcards app, focusing on gamified, mechanical recognition over realistic, spoken conversations.

Only in recent years, paying for Duolingo Max gets you two AI-powered learning modes. Its Video Call mode can feel too open-ended, leaving you practicing random chit-chat, while its Roleplay mode still seems to fall short on targeted, actionable, scenario-aware feedback, especially on pronunciation and fluency.

With these issues in mind, we asked ourselves: What would a voice-first language learning app built around real-world conversations look like?

Idea: Voice-First Language Learning

As part of SMU’s Master of IT in Business programme), my team experimented with the design and development of a language learning app prototype named Converse AI. By leveraging Large Language Models (LLMs), it aimed to teach language naturally through dynamic voice conversations in practical scenarios, such as ordering coffee at a cafe or asking strangers for directions. The prototype supported both English and Mandarin Chinese.

This is how a language learner would use Converse AI:

(1) Choose a Scenario

You first select the language they want to practice and chose a scenario from a preset list of everyday situations, such as “Ask for directions”, “Report lost passport” and “Order food at a restaurant”.

(2) Have a Conversation

Credits to 長井保 for the Live2D animated avatar model.

Next, you get dropped into a voice conversation with an AI conversation partner that simulates natural dialogue in the chosen scenario. The partner adopts a relevant persona — such as being a helpful local resident, an embassy official, or a restaurant waiter — and responds to your speech in an adaptive, non-scripted manner. While you use push-to-talk to capture your speech, the partner delivers their speech through real-time audio, accompanied by text subtitles and an animated avatar.

(3) Learn from Feedback

After ending the conversation, you receive a holistic assessment of how well you spoke. The assessment includes both numerical scores and descriptive feedback that highlights specific strengths and areas for improvement. You can also read targeted and actionable feedback for each of your responses during the conversation, such as “‘I want’, not ‘I wants’”. A curated list of words that you likely mispronounced is shown as well, allowing you to learn from your mistakes.

Here’s a short video demo:

[embed]

Under the Hood

The Converse AI prototype frontend was built with React and Live2D Cubism (for the animated avatar). It maintains a low-latency, bidirectional WebSocket connection to the backend, which streams the LLM responses and audio in real time. Its backend is a FastAPI server that orchestrates all the external API calls and data processing logic:

System architecture for the Converse AI prototype

• For conversation generation and feedback, the backend relies on either LLMs running on Ollama (for locally-hosted LLMs) or the Gemini API.
• For real-time speech transcription, the Python runtime loads a custom fine-tuned Whisper speech model via HuggingFace’s transformers library
• For text-to-speech generation, the backend integrates with Microsoft’s edge-tts

In the design of this LLM application, it was helpful to distinguish between two distinct AI pipelines, each with different objectives and considerations: Conversation Practice and Conversation Feedback.

Conversation Practice

AI pipeline powering the conversations

Given a real-world scenario and AI partner persona, the Conversation Practice AI pipeline aims to maximize the Realism, Coherence and Conciseness of the AI partner’s dialogue in a conversation while minimizing latency and cost:

• Realism assesses whether responses are believable, grounded, and free from implausible claims.
• Coherence measures how well responses stay on topic and maintain logical flow.
• Conciseness looks at whether ideas are conveyed clearly and directly without unnecessary words or repetition.

To choose the best model, we benchmarked several LLMs on a curated subset of MT-Bench from the “Roleplay” category on the above dimensions. Our comparison covered a diverse mix of small open-weight architectures that can be run locally on consumer GPUs, as well as a frontier, proprietary API-based LLM. Across both English and Mandarin Chinese, we found that a simple self-hosted Qwen 2.5 7B could produce surprisingly high-quality responses, especially on coherence and realism. Nevertheless, we concluded that Gemini 2.5 Flash Lite is still the most practical choice for such an application due to its comparable performance, cost-effectiveness and lower latency, without any need to manage local GPU resources.

Conversation Feedback

AI pipeline powering the generation of conversation feedback

Given a conversation transcript with a learner, the Conversation Feedback AI pipeline aims to maximize the accuracy of the written feedback and analysis of the learner’s language proficiency.

We evaluated its accuracy based on Complexity, Accuracy, and Fluency (CAF), a widely recognized linguistic model for evaluating language proficiency:

• Complexity: The range and sophistication of grammar and vocabulary used.
• Accuracy: The correctness of grammar, word choice, and pronunciation.
• Fluency: The smoothness and pace of speech, including pauses, fillers, and disfluency patterns.

Everything in this feedback pipeline depends on transcribing the learner’s speech accurately. However, we found that off-the-shelf Whisper by OpenAI, popular for Automatic Speech Recognition (ASR), tends to over-smooth transcripts by removing pauses, fillers, and word-level errors. To preserve these signals, we fine-tuned Whisper with Low-Rank Adaptation (LoRA) on learner-style speech that intentionally includes disfluencies, repetitions, and grammatical mistakes. We built this training corpus for both English and Mandarin Chinese by prompting Gemini Pro 2.5 to generate realistic transcripts, then using Google Text-to-Speech to convert them into audio for fine-tuning. Our fine-tuned Whisper model achieved an 80.8% reduction in Word Error Rate (WER) on English learner speech compared to the original Whisper, and a 43.9% reduction in Character Error Rate (CER) for Mandarin Chinese learner speech.

We supplemented ASR with acoustic and linguistic features, using OpenSMILE and librosa for prosodic and spectral signals, and spaCy for structural language metrics such as sentence complexity and lexical diversity. We also integrated Microsoft Azure Speech API for word- and phoneme-level pronunciation scoring.

In the final stage, an LLM turns all of these signals into a CAF report that is shown to the learner after the conversation. The pipeline feeds a consolidated context bundle into the feedback model: the ASR transcript, acoustic features, linguistic metrics, pronunciation scores, and a detailed CAF rubric in the prompt. We found that Gemini 2.5 Flash Lite was consistently capable of generating detailed, well-structured CAF feedback, and that feedback is strongly aligned with the scores provided by human evaluators across English and Mandarin Chinese.

The Reality Check

The Converse AI prototype worked, but it was far from perfect. The fine-tuned ASR model relies too heavily on synthetic learner speech, pronunciation scoring is not yet robust enough, and our human-evaluation sample remains limited. Even with those caveats, the prototype shows that a voice-first language learning experience, powered by LLMs, is not only technically feasible, but can also provide a more realistic and engaging way for learners to practice and improve their spoken language skills compared to existing solutions.

Special thanks to my teammates who made this project possible: Wyman, XiongQing, Justin and Mikael.

All opinions and interpretations are that of the writer, and not of MITB. I declare that I have full rights to use the contents published here, and nothing is plagiarized. I declare that this article is written by me and not with any generative AI tool such as ChatGPT. I declare that no data privacy policy is breached, and that any data associated with the contents here are obtained legitimately to the best of my knowledge. I agree not to make any changes without first seeking the editors’ approval. Any violations may lead to this article being retracted from the publication.