Why Speech AI Keeps Failing in the Real World And What Actually Fixes It

Why Speech AI Keeps Failing in the Real World And What Actually Fixes It

Open any major voice assistant on your phone. Ask it something in a noisy room. Speak with a regional accent. Use a word that’s common in your language but rare in English. Watch it fail.

This isn’t a model problem. The models are capable. It’s a data problem — specifically, a training data problem that has been papered over for years by benchmarks that measure performance in conditions that don’t reflect how people actually talk.

The speech AI industry has a dirty secret: most of the training data it runs on was collected in studios, by native English speakers, reading carefully from scripts, in quiet rooms with professional microphones. The models trained on this data perform extraordinarily well on benchmarks designed to test exactly those conditions. They perform poorly everywhere else.

Fixing this requires going back to the source. It requires building data collection infrastructure that captures speech the way it actually sounds — in real environments, from real speakers, with real acoustic variation, in real languages. Not just English. Not just studio-clean. Not just scripted.

This is what we’ve built in Kathmandu.

The Training Data Problem in Speech AI

To understand why the data problem is so persistent, it helps to understand the economics of speech data collection.

High-quality speech data is expensive to collect properly. A controlled recording environment, calibrated microphones, trained speakers, precise transcription, quality review — the cost per hour of properly collected speech data is significantly higher than most other data types. And the volume required for robust model training is enormous. State-of-the-art ASR systems train on tens of thousands of hours of data. TTS voice banks require hundreds of hours per speaker style.

The industry response to this cost problem has been to cut corners in ways that aren’t immediately visible in benchmark results. Crowdsourced recordings from consumer smartphones. Gig workers reading scripts in their apartments. Synthetic data generated from existing TTS systems, used to train better TTS systems in a circular dependency that compounds existing biases. Web-scraped audio from podcasts and YouTube, with automated transcription that introduces systematic errors.

None of these approaches is necessarily wrong in isolation. All of them, at scale, produce training datasets that are systematically biased toward conditions that don’t reflect real-world deployment — and models that fail precisely when users need them most.

The failure modes are predictable and consistent. Models underperform on accented speech because their training data was dominated by native speakers. They fail in noisy environments because their training data was clean. They struggle with spontaneous conversation because their training data was scripted. They perform poorly on low-resource languages because their training data was overwhelmingly English.

These aren’t edge cases. For a significant portion of the world’s population — including the 1.4 billion people whose first language is not English — these failure modes represent the normal experience of interacting with voice AI.

What Training-Grade Speech Data Actually Requires

The gap between “audio recordings” and “training-grade speech data” is larger than most non-specialists realize.

Acoustic quality is the baseline. Clean recordings with controlled noise floors, consistent microphone placement, and adequate bit depth and sample rate are the minimum requirement — not because real-world deployment is acoustically clean, but because the training pipeline needs clean source material that can be systematically augmented with realistic noise profiles. Noisy source recordings introduce uncontrolled artifacts that models learn as signal rather than noise.

Speaker diversity is where most datasets fail. A model that has seen ten thousand hours of speech from a demographically narrow set of speakers will generalize poorly to speakers outside that set. Robust ASR requires coverage across age, gender, accent, dialect, speaking rate, and vocal quality. Robust TTS requires the same diversity in the voice banks used for training.

Linguistic coverage matters beyond language. Within a single language, there are significant variations in vocabulary, syntax, and pragmatics across regions, demographics, and contexts. A model trained on formal, written-influenced speech will struggle with the vocabulary and rhythm of casual conversation. A model trained on urban speech will struggle with rural vocabulary and accent. These distinctions require intentional collection design, not just volume.

Annotation quality is the final and most frequently compromised requirement. Transcripts that are 95% accurate sound acceptable but introduce systematic errors that scale with dataset size. Speaker diarization that misattributes turns corrupts conversational training data. Intent labels that are inconsistently applied undermine fine-tuning quality. Training-grade annotation requires human review at every stage, not automated processing with spot checks.

Each of these requirements adds cost. Each of them also adds value — and the value compounds, because high-quality data produces higher-quality models that require less data to reach a given performance threshold.

What We’ve Built in Kathmandu

FileMarket AI’s speech data collection center in Kathmandu is purpose-built for training-grade audio dataset production. It is not a gig platform. It is not a crowdsourcing operation. It is a controlled, supervised, in-house facility where every session is monitored for quality, every audio file is reviewed before annotation, and every transcript is human-verified before delivery.

The facility runs on professional Logitech headset microphones — chosen for consistent acoustic properties across units, comfortable for extended sessions, and deployable at scale without per-unit calibration overhead. The recording environment is controlled for background noise. The session management system ensures consistent microphone placement and recording parameters across speakers and sessions.

Our collection capability spans four primary data types.

ASR training data covers the full spectrum that robust speech recognition requires: read speech from carefully designed prompt sets, spontaneous conversational speech with natural disfluencies and turn-taking, command and control utterances for voice interface training, and environmental recordings that capture the acoustic variation of real deployment conditions. Each recording type is designed to address a specific failure mode in models trained on studio-only data.

TTS voice banks are collected with the tonal and stylistic diversity that makes synthesized speech genuinely human-sounding. Neutral, expressive, whispering, fast-paced, deliberate — multiple speaking styles from each speaker, enabling fine-grained control of synthesized output. Multi-speaker datasets with careful demographic coverage ensure that synthesized voices don’t default to a narrow demographic profile.

LLM fine-tuning data captures conversational speech with the structural annotation that language model training requires — turn-level transcription, speaker diarization, intent labeling, and emotional tone annotation. This is the data that enables voice-first LLM interfaces to understand not just what was said but how and why.

Nepali language data is a specific focus area. Nepali is one of the most underrepresented languages in AI training data relative to its speaker population. Collecting high-quality Nepali speech data in-house, with native speakers, in a controlled environment, produces a resource that simply doesn’t exist at scale anywhere else. For AI systems being deployed in Nepal and the broader South Asian region — increasingly a priority market for voice AI — this is the data that makes the difference between a system that works and one that doesn’t.

The In-House Advantage

The difference between in-house collection and crowdsourced or gig-based alternatives is not primarily about quality per recording — it’s about consistency, control, and the ability to respond to specific data requirements.

When a client needs speech data with specific acoustic properties, specific speaker demographics, specific linguistic coverage, or specific annotation schemas, in-house collection can be designed and executed to those specifications. Crowdsourcing produces whatever the crowd produces. In-house production produces what the specification requires.

This matters particularly for edge cases — the conditions where models fail and where targeted data collection can fix specific failure modes. A client whose ASR model struggles with a specific accent can commission targeted collection of that accent. A client whose TTS system sounds unnatural in a specific speaking style can commission targeted voice bank expansion. These are conversations that require a production facility, not a platform.

The cost structure of in-house collection in Kathmandu also allows us to offer training-grade quality at cost points that would be unachievable in higher-cost labor markets. This is not a compromise — it is a structural advantage that allows our clients to collect more data, at higher quality, for equivalent budget.

Where Speech AI Needs to Go

The next generation of voice AI — models that can understand speech in genuinely diverse real-world conditions, synthesize voices that are indistinguishable from human, and converse naturally in hundreds of languages — will be trained on data that looks nothing like the studio recordings that underpin current state of the art.

It will be trained on data collected in real environments, from real speakers, with real linguistic diversity, annotated with the precision that production models require. It will include languages that have historically been ignored because the collection infrastructure didn’t exist. It will cover acoustic conditions that current benchmarks don’t test because the training data to support them hasn’t been collected.

Building that data is infrastructure work. It requires facilities, equipment, speakers, annotators, quality processes, and the operational discipline to run them consistently at scale. It is not glamorous. It does not generate the kind of attention that a new model architecture or a benchmark result does.

But it is the foundation that everything else runs on.

We’re building it in Kathmandu.

Learn more about FileMarket AI Data Labs: https://filemarket.ai

Speech data inquiries: humanloop@filemarket.ai