From Chaos to Clarity: How Indic Text Normalization Transforms Speech Recognition

Indic Validation is the process of validating content written in Indic languages from a script grammar point of view. It validates if the composition of a word is correct as per the rules of the script.

Indic Normalization is the process of converting different written forms of an Indic word to a single standardized form.

Spell Variants Normalization is the process of automatically identifying misspelled words and replacing them with the proper spelling. This is done leveraging the phonetic properties of Indic languages.

What’s the need?

Digitally sourced Text data is noisy primarily due to

• Typing Tools allowing users to type any combination (No Standard Tool with proper guidelines)
• Lack of Proper Spell Checker tools for Indic Languages
• The way of writing by hand in a notebook is different from text written digitally. eg: काेला is written mostly keeping in mind 5 parts [ क + ा + े + ल + ा ], in actual should be 4 parts.

Impact of Noisy Text on Speech Recognition:

• Language Model (LM) Degradation ASR systems often use a language model to predict likely word sequences. If the LM is trained on noisy text, it may learn unrealistic or incorrect patterns.
• Pronunciation Errors Noisy or inconsistent text can mislead pronunciation modeling. This is particularly problematic when using grapheme-to-phoneme (G2P) models. eg: if आम is written as अ + ा + म, our pronunciation module will output AH AA M, instead of AA M. Unwanted tokens get generated due to wrong data
• Evaluation Bias If ground-truth transcriptions are noisy, Word Error Rate (WER) and Character Error Rate (CER) will be misleading. You might get penalized for “correct predictions” just because the reference is wrong.
• Mismatch with Acoustic Data Noisy text can introduce label noise, causing misalignment between audio and text. This degrades the performance of acoustic models during training.

Examples:

Where’s the error:

Reference : अमेज़न खोलो ASR output: अमेज़न खोलो

Reference : कोका कोला चाहिए ASR output: कोका काेला चाहिए

Reference : हिंदी प्रतिनिधि से जोड़िए ASR output: हिन्दी प्रतिनिधि से जोड़िये

For humans, all are 100% correct, but for machines it’s not.

Here’s the error:

Reference : अमे ज़ न खोलो ASR output: अमेज ़ न खोलो Error = (S+D+I)/W = (1+0+0)/2 = 50%

Reference : कोका क ो ला चाहिए ASR output: कोका क ा े ला चाहिये Error = (S+D+I)/W = (2+0+0)/3 = 66.66%

Reference : हिंदी प्रतिनिधि से जोड़िए ASR output: हिन्दी प्रतिनिधि से जोड़िये Error = (S+D+I)/W = (2+0+0)/4 = 50%

With Indic Normalization and Validation, all the 3 examples will give 0% error.

We did an exercise on Shrutilipi Hindi language dataset and performed Normalization, Validation and Spell Variants removal:

Total sentence count: 734675 Invalid sentence count: 159867

Total unique words=146891 Total unique words after Normalization=143047 Unique invalid words before normalization = 9149 Unique invalid words after normalization = 2865