what telephone degradation does to hindi whisper models

18,000 recordings. 3,000 training steps. A 10-point WER improvement. And a model that failed the test that mattered.

let me define the terms in the title first:

telephone degradation -> audio has a sampling rate, a bit like video has a frame rate. a higher sampling rate lets us capture a wider range of frequencies.

a traditional telephone channel samples audio at 8 kHz, which is quite low compared with other forms of recording. even at 8 kHz, the Nyquist-Shannon sampling theorem tells us that we can only represent frequencies up to 4 kHz.

the missing frequencies contain useful speech details, especially for consonant sounds. converting an 8 kHz recording into a 16 kHz file gives Whisper the correct input format, but it does not bring those details back. the model receives a valid audio file with less information inside it.

upsampling the telephone version back to 16 kHz changes the file format. the missing frequencies remain missing.

so why do we still use 8 kHz in telephony?

because telephone networks were designed to transmit understandable speech efficiently, not to preserve the full quality of a human voice.

most of the frequency information required to understand a conversation lies within a relatively narrow range. traditional telephone systems preserve roughly 300 Hz to 3.4 kHz. an 8 kHz sampling rate is enough to carry this range while keeping the amount of transmitted data low.

this mattered when telephone infrastructure was being developed. lower bandwidth meant that more calls could travel through the same network, with lower transmission and storage costs. standards and codecs such as G.711 were built around these constraints, and 8 kHz remained common across traditional phone networks and call-centre recordings.

humans can usually understand this audio because we use context to fill in missing details. an ASR model works differently. it has learned relationships between acoustic patterns and words from its training data. but when the telephone channel removes frequency information or introduces codec distortion, those patterns change.

this leads to the question behind this experiment:

do different Hindi Whisper models lose the same amount of accuracy when the exact same speech is passed through a telephone channel?

what are Hindi Whisper models?

Whisper converts spoken audio into text and was trained on multilingual speech, including Hindi.

the original Whisper models can recognise Hindi, but they are general-purpose models. a model can be further trained on Hindi speech so that its weights become better adapted to Hindi pronunciation, vocabulary, and speaking patterns. the saved result of this additional training is often called a fine-tuned checkpoint.

i primarily compared two Hindi-tuned checkpoints: ARTPARK Whisper-medium and Adalat Whisper-small.

both are based on Whisper, but they differ in model size and training data. ARTPARK uses the larger Whisper-medium architecture, while Adalat uses Whisper-small and requires less computation during inference.

that gives us the comparison i care about.

the smaller model is faster, but does it also lose more accuracy when the same speech is passed through a telephone channel?

how is speech recognition accuracy measured?

it is dead simple. WER is based on the minimum number of edits needed to turn the model transcript into the reference:

WER = (substitutions + deletions + insertions) / reference words

where:

• substitution: the model predicts the wrong word.
• deletion: the model misses a reference word.
• insertion: the model adds a word that was never spoken.

a WER of 20% roughly means one word error for every five reference words. WER can exceed 100% if the model inserts many additional words.

WER is imperfect. it treats every word edit equally, even though confusing a person’s name may matter more than missing a filler word.

there’s also CER, which performs the same calculation at the character level and helps reveal smaller spelling differences. for the main comparisons, however, we use WER.

GramVaani

Gram Vaani is an Indian social technology company behind Mobile Vaani, a phone-based community platform. people can call from a basic phone, without internet, to record messages or listen to what others have posted. some of those calls were later transcribed and released as the GramVaani speech dataset.

that gives us real spontaneous Hindi telephone speech instead of sentences recorded in a studio. it also gives us regional accents, background noise, inconsistent recording quality and messy transcripts. exactly the kind of audio that makes ASR difficult.

i started with this dataset. the initial result appeared to confirm that 8 kHz audio was the main problem, but the comparison had an important flaw.

we first tested on 50 recordings, of which 32 were 8 kHz recordings. Whisper-small produced 92.39% WER on the native 8 kHz group and 70.03% on the higher-rate group. this looked like a telephone-channel penalty of more than 22 WER points… at first.

upon digging deeper though, i found that the two groups did not contain the same speech. they had different speakers, gender distributions, topics, noise levels, and recording conditions. any of these differences could have contributed to the higher WER.

to measure the effect of the channel itself, we needed to keep the speaker, utterance, and transcript fixed while changing only the audio condition.

building a fair comparison

so i did the obvious thing: take one recording and create several versions of it.

• the original recording
• audio limited to an 8 kHz telephone channel
• 8 kHz audio passed through G.711 A-law
• 8 kHz audio passed through G.711 μ-law
• 8 kHz audio passed through GSM

the codec versions were band-limited first because that is closer to what happens during a telephone call. all the final files were converted back to 16 kHz before being given to Whisper, since that is the input rate Whisper expects.

now the speaker, words, accent, microphone and background noise remain the same. only the audio channel changes. both models also receive the same files with the same transcription settings.

this gives us a paired comparison. instead of comparing two unrelated groups of recordings, we can measure how much each model gets worse on a degraded copy of the same speech.

we call this difference the channel penalty:

channel penalty = WER on degraded audio − WER on original audio

the larger the penalty, the more accuracy the model loses because of the telephone transformation.

so what did we find?

now we can return to the original question without mixing up different speakers and recordings: do both models lose the same amount of accuracy?

i first ran this paired comparison on Vaani, which is a different dataset from GramVaani. i selected one recording from each of 500 speakers and created all five versions of every recording.

ARTPARK lost less than half a WER point. Adalat lost 2.80 points on the exact same speech transformations. the difference between their channel penalties was 2.32 points.

but how do we know that this was not caused by a few unusual speakers?

i used a bootstrap test, which basically repeats the calculation thousands of times with different samples of speakers. after 20,000 repetitions, the estimated gap remained between 1.66 and 3.00 WER points in 95% of the samples. since this range stayed above zero, the difference was unlikely to be random.

there was still another problem. ARTPARK had already been trained on Vaani data, so it might have been unusually comfortable with this dataset.

to check whether the result was specific to Vaani, i repeated the same experiment on LAHAJA using one recording from each of its 132 speakers.

LAHAJA made the tradeoff easier to see. Adalat was slightly better on the original recordings, but became worse after telephone degradation. its channel penalty was 2.92 points larger than ARTPARK’s.

so the same pattern appeared on two different Hindi datasets. Adalat Whisper-small was faster, taking around 53% of ARTPARK’s inference time, but ARTPARK Whisper-medium was consistently less affected by the telephone channel.

these are the exact channel penalties across the two datasets:

the smaller Adalat model suffered a much larger pooled channel penalty on both external datasets.

the smaller Adalat model suffered a much larger pooled channel penalty on both external datasets.

can we train the smaller model to handle this better?

Adalat was the obvious model to work on. it was almost twice as fast, and on the original LAHAJA recordings it was even slightly more accurate than ARTPARK. its weakness only became clear after the telephone transformations.

so the next question was: can we keep the speed and reduce the channel penalty?

for this, i used LoRA. instead of updating every weight in Whisper-small, LoRA freezes the original model and trains a much smaller set of additional parameters. this makes the experiment possible on a Colab T4 without training the whole model from scratch.

the adapter was trained on 18,000 GramVaani recordings. every selected source clip kept its released telephone version, while one-third received an extra 8 kHz, G.711 or GSM version. the final mix was roughly 75% released GramVaani audio and 25% added channel stress, giving about 65 hours of training views over 3,000 steps.

Vaani and LAHAJA were kept out of training. even the internal GramVaani split was separated by recording group, so the same recording could not appear on both sides.

and initially, it worked:

that is a drop of more than ten WER points on both the released audio and the telephone conditions. on the internal test, the adapter looked clearly better.

but both the training data and this test still came from GramVaani. the model had learned something useful there. we still did not know whether it had learned general telephone robustness or simply become better at GramVaani.

did the improvement survive outside GramVaani?

short answer: no. the adapter had overfit to GramVaani.

• on Vaani, pooled telephone WER became worse: 20.22% → 22.62%.
• on LAHAJA, it barely changed: 21.53% → 21.89%. so there was no improvement there either.
• original-audio WER also became worse on both datasets.

the adapter had learned something specific to GramVaani, such as its speaking style, vocabulary, recordings, or transcription patterns.

why did the adapter fail?

the adapter had seen 18,000 GramVaani recordings, but they still came from one dataset. the extra G.711 and GSM versions made the audio rougher. they did not add new accents, topics, microphones or transcription styles.

even the internal test came from GramVaani. separating recording groups prevented the same recording from appearing in both training and evaluation, but both sides still shared the same source and transcript conventions.

the problem was what the model was being rewarded for.

its job was to reduce transcription error on the GramVaani training mix. nothing explicitly forced it to produce the same transcript for an original recording and its degraded copy. it could lower the loss by becoming better at GramVaani without learning general telephone robustness.

that is the most likely explanation for the final result: a ten-point improvement inside GramVaani, worse performance on Vaani, and no improvement on LAHAJA.

a second attempt would need more variety. i would use Hindi and Hinglish speech from several datasets, mix clean and real telephone recordings, and train on matched original/degraded pairs. model selection would also need to use more than one domain.

Vaani and LAHAJA cannot be the final test again. we have already seen their results, so tuning against them would turn them into validation sets. a new attempt needs a new untouched benchmark.

what this project shows

once i tested the adapter on Vaani and LAHAJA, i could no longer call it an improvement.

i am publishing the training run as a failed adaptation case, along with its setup and evaluation results. the benchmark caught a problem that the internal GramVaani score hid.

the paired benchmark produced the result i trust most from this project. Adalat’s channel penalty was 2.32 WER points larger on Vaani and 2.92 points larger on LAHAJA. it also took around 53% of ARTPARK’s inference time.

Adalat is faster, but it lost more accuracy when the same speech was passed through the same telephone transformations. that pattern survived a second dataset.

where this result stops

the 8 kHz, G.711 and GSM conditions are controlled simulations. they isolate specific kinds of channel damage. real calls can also have packet loss, clipping, network jitter and device noise, none of which were covered here.

the absolute WER values across GramVaani, Vaani and LAHAJA should not be compared directly either. Vaani uses multiple human references, while GramVaani and LAHAJA use one. the useful comparisons happen inside each dataset.

ARTPARK had also seen Vaani during training. that could help its Vaani score, which is why repeating the result on LAHAJA mattered.

and this is still a comparison between two specific checkpoints. it does not mean every Whisper-small model will handle telephone audio worse than every Whisper-medium model.

the answer to the question in the title (atleast rn): these two Hindi Whisper models did not react to telephone degradation in the same way. the smaller model was faster, but more sensitive to the channel.

training reduced its error inside GramVaani, then failed to carry that improvement anywhere else.

I’ve published these experiments here:

https://github.com/anshulLuhsna/CallWhisper-8k