The Real Failure Mode of Multilingual LLMs: Boundary Errors, Not Language

Structured extraction is where things actually break in production — not loudly, but in ways that are easy to miss. The model doesn’t hallucinate something obviously wrong. It gives you something almost right. That’s worse, because it slips through and quietly corrupts downstream logic.

I kept running into this in real workflows — mixed-language inputs, messy phrasing, transliteration — and the model would confidently extract the wrong value or just miss one. It didn’t feel like a language understanding issue. Most of the time, the intent was clearly understood. The failure was in the last mile — getting the exact entity right.

That’s what pushed me to isolate it. I stripped things down to a controlled setup: same task, same schema, only varying how the input is written to see what actually breaks.

Most people frame multilingual LLM performance as a language problem — can it handle Hinglish, Telugu-English, mixed scripts, etc. I assumed the same going in. I tested models including ones from Sarvam AI, which are explicitly built for Indian languages, expecting them to handle this better.

What showed up was… not that.

The failures weren’t really about understanding the language. The models usually got the intent right. Where things broke was in extraction — missing entities, drifting values, or just cutting boundaries wrong.

Language wasn’t the bottleneck.

Extraction was.

One of the reasons I included Sarvam AI in this setup is because they’re actually going after a real problem — handling Indian languages the way people actually use them. Most systems still assume clean, well-formed input. That’s not reality.

Sarvam’s focus on native languages and code-mixed usage makes them one of the few teams building for that messiness instead of ignoring it.

Experiment Setup

I kept the setup deliberately simple. The goal wasn’t to optimize prompts or chase best-case performance — it was to see how models behave when the input gets messy.

Task Structured extraction. Given an input, extract a fixed set of fields. Same schema across all runs. No flexibility — the output either matches or it doesn’t.

Input Types Two categories:

• Hinglish (Hindi + English in Latin script)
• Telugu-English (code-mixed, often transliterated)

Examples were left as-is. No cleanup, no normalization.

Models Tested - sarvam-30b (multilingual, Indian language-focused)

• gpt-4.1-nano (smaller, general-purpose)

Constraints - Same prompt across all runs

• Same schema
• Single pass (no retries)
• No post-processing

This was intentional. I didn’t want system design masking model behavior.

Evaluation Each output falls into one bucket:

• Correct — all entities extracted accurately
• Partial — intent correct, but one or more entities wrong or missing
• Failure — wrong intent or unusable output

No gray area.

Where Things Break

First observation — this is not a language understanding problem.

Across both Hinglish and Telugu-English inputs, models usually got the intent right. They knew what the user was asking. The failure shows up one step later — when converting that into structured output.

Hinglish: Mostly Partial Failures Hinglish rarely caused full breakdowns. Instead, outputs were close — but not clean.

Typical pattern:

• Right intent
• Some entities correct
• One entity slightly off or missing

These are “almost usable” outputs — which is exactly what makes them risky.

Telugu-English: More Hard Failures Telugu-English behaved differently.

Failures were more likely to be complete:

• Missing outputs
• Wrong intent
• Entire extraction unusable

This suggests the model struggles earlier when parsing becomes harder.

The Common Failure: Boundary Errors Across both, one pattern dominates — boundary mistakes.

The model knows what to extract. It fails at:

• where the entity starts
• where it ends
• what belongs inside it

That shows up as:

• truncated values
• merged entities
• shifted fields

These aren’t random errors. They’re systematic.

Language isn’t the bottleneck.

Extraction precision is.

What the Numbers Say

Once you bucket the outputs, the pattern is clear.

The dominant failure mode:

• Intent correct, entity wrong or missing (~40%)

This is the dangerous category. Looks fine at a glance, but breaks downstream.

By input type:

Hinglish → more partial failures

• High entity drift
• Missing or slightly wrong values
• Rarely completely broken

Telugu-English → more full failures

• More null outputs
• More wrong intent cases
• Entire extraction unusable

Two more observations:

• Prompt language had negligible impact Changing prompt language didn’t shift outcomes meaningfully.

• Smaller model was more stable gpt-4.1-nano was more consistent. sarvam-30b showed higher variance across similar inputs.

Not better at language. Just more predictable in extraction.

What This Actually Means

If you’re building systems that depend on structured extraction, this changes where the problem is.

It’s not about whether the model understands the language.

It’s about whether it can translate that understanding into clean, bounded entities.

That’s a different failure class.

And it’s easy to miss, because nothing looks obviously wrong. The output reads fine. The intent is correct. But the extracted values are just off enough to cause issues downstream.

Implications:

• Treat extraction as a precision problem, not a language problem
• Assume partial failures are common, not edge cases
• Don’t trust “looks correct” — validate at the entity level

Also, better multilingual models alone won’t fix this. The failure sits at the boundary between understanding and structuring — not purely in either.

If anything, this strengthens the case for post-generation validation layers, especially in multilingual settings where ambiguity and transliteration amplify small mistakes.

The takeaway: Language understanding gets you close. Extraction precision determines if it actually works.

Code and Reproducibility

All code, evaluation scripts, and outputs are available here:

https://github.com/bh3r1th/sarvamai-llm-failure-map

The repository includes:

• dataset for multilingual extraction evaluation
• model outputs across Hinglish and Telugu-English inputs
• evaluation scripts for bucketing (correct / partial / failure)
• aggregate metrics by model, language, and failure type
• commands to reproduce the full experiment

This is a focused, experiment-first repo — built to make the failure patterns easy to inspect, not a generalized framework.

Disclaimer: The posts here represent my personal views, not those of my employer or any specific vendor. Any technical advice or instructions are based on my knowledge and experience.