Building an Offline Indian Languages Speech-to-Text iOS App with AI4Bharat and CoreML

How I converted a fairseq/HuggingFace wav2vec2 model to run entirely on-device on iPhone — and the two invisible bugs that only showed up after the Python validation had already passed.

The goal

I wanted a SwiftUI app that transcribes 22 official Indian languages to text without a network connection — no API calls, no server round-trip, everything running on the phone’s own Neural Engine/GPU/CPU. The obvious source for a Hindi ASR model is AI4Bharat, a research group building open speech and NLP models for Indian languages.

The catch: AI4Bharat ships PyTorch/fairseq checkpoints. There is no CoreML version, no iOS SDK, no tutorial for “how do I put this on an iPhone.” That gap — HuggingFace model to a working SwiftUI app — is what this post is about.

Picking a model

AI4Bharat has two broad ASR families worth considering:

For an offline, on-device MVP, size and inference speed mattered more than squeezing out the last few points of accuracy, so I went with [ai4bharat/indicwav2vec-hindi](https://huggingface.co/ai4bharat/indicwav2vec-hindi) — a Wav2Vec2ForCTC-style model, originally trained in fairseq and ported to HuggingFace transformers format, Apache 2.0 licensed (though the HF repo itself is gated, so you need to accept the terms and generate an access token before you can download it).

Scope for v1: For trying i have used Hindi, tap-to-record-then-transcribe (not live streaming captions — more on why below).

The conversion pipeline

The path from HuggingFace to iOS is: PyTorch → traced graph → CoreML .mlpackage, using coremltools.

1. Load the model

python from transformers import AutoModelForCTC model = AutoModelForCTC.from_pretrained(“ai4bharat/indicwav2vec-hindi”) model.eval()

Nothing exotic — this is the standard transformers loading path.

2. Wrap it for tracing

The raw model’s forward() returns a Wav2Vec2CTCModelOutput object with several named fields. CoreML’s PyTorch tracer wants a plain tensor in, tensor out, so I wrapped it:

```python class CTCLogitsWrapper(torch.nn.Module): def init(self, model): super().init() self.model = model

*def forward(self, input_values): return self.model(input_values).logits

**3. Trace it — deliberately without an attention mask**

*```python
dummy_input = torch.randn(1, 16_000 * 10) # 10 seconds of fake audio
with torch.no_grad():
 traced = torch.jit.trace(wrapped, dummy_input, strict=False)
```*

This is a small but important decision. `*Wav2Vec2Model`’s* attention implementation has a shape-dependent branch that checks the attention mask against the input length. If I passed a mask in, `torch.jit.trace` would bake in that specific control-flow path as if it were a constant — which works for the traced length but silently breaks (or worse, doesn’t error, just produces wrong output) for other lengths. Since the app only ever processes one un-padded utterance at a time (batch size 1, no padding needed), the mask is never actually necessary, so I just… didn’t pass one. That sidesteps the whole problem.

**4. Convert with a flexible input shape**

*```python
import coremltools as ct
import numpy as np*

*mlmodel = ct.convert(
 traced,
 inputs=[
 ct.TensorType(
 name=”input_values”,
 shape=(1, ct.RangeDim(16_000, 480_000)), # 1–30 seconds @ 16kHz
 dtype=np.float32,
 )
 ],
 outputs=[ct.TensorType(name=”logits”, dtype=np.float32)],
 minimum_deployment_target=ct.target.iOS16,
 compute_precision=ct.precision.FLOAT16,
 convert_to=”mlprogram”,
)
```*

The `*RangeDim(16_000, 480_000)*` is what lets **one** compiled model accept anywhere from 1 to 30 seconds of audio, instead of needing separate models baked for fixed durations. I didn’t just trust that this worked because the API accepted it — I tested it directly:

*```python
for seconds in [2, 3, 15, 25]:
 audio = np.random.uniform(-0.1, 0.1, size=16_000 * seconds).astype(np.float32)
 out = mlmodel.predict({“input_values”: audio.reshape(1, -1)})
 print(seconds, “->”, list(out.values())[0].shape)
```*

All four lengths produced correctly-shaped output, confirming the flexible dimension actually works end to end rather than just at the traced length.

*`compute_precision=FLOAT16*` roughly halves the model size — the converted `*.mlpackage*` came out to* ~602MB*, down from what would have been well over 1GB in float32.

**5. Export the vocabulary separately**

The model’s tokenizer is Devanagari, character-level. Rather than hardcoding guessed token IDs into the Swift app (a classic source of silent, hard-to-debug mismatches), I exported it directly from the loaded tokenizer:

*```python
from transformers import Wav2Vec2CTCTokenizer
tokenizer = Wav2Vec2CTCTokenizer.from_pretrained(“ai4bharat/indicwav2vec-hindi”)
vocab = {
 “id_to_token”: {v: k for k, v in tokenizer.get_vocab().items()},
 “blank_id”: tokenizer.pad_token_id,
 “word_delimiter”: tokenizer.word_delimiter_token,
}
```*

(One gotcha here: the repo’s default `AutoProcessor` resolves to `Wav2Vec2ProcessorWithLM`, which requires the `pyctcdecode`/`kenlm` dependencies for n-gram beam-search decoding. Since the app only does greedy CTC decoding — no language model — I loaded the plain `Wav2Vec2CTCTokenizer` directly and skipped that dependency entirely.)

This produced a 68-token `vocab.json` — Devanagari characters plus special tokens (`<pad>` as the CTC blank at id 0, `|` as the word delimiter, `<unk>`, `<s>`, `</s>`).

**6. Validate — compare PyTorch vs. CoreML on the same input**

*```python
torch_logits = model(input_values).logits[0].numpy()
coreml_logits = mlmodel.predict({“input_values”: audio})[“logits”][0]*

*argmax_agreement = np.mean(
 np.argmax(torch_logits, axis=-1) == np.argmax(coreml_logits, axis=-1)
)
```*

This came back at **100% argmax agreement**, with a max absolute logit difference of about 0.02 (expected float16 rounding noise). Conversion confirmed correct — or so I thought.

## Two bugs that Python validation couldn’t catch

This is the part of the process worth remembering more than any of the code above: **the Python-side validation passed cleanly, and the app still crashed on a real device.** Both bugs came from the same root cause — `*coremltools.predict()*` in Python quietly normalizes its output into convenient numpy arrays, which papers over what’s actually stored on-device.

**Bug 1: “No space left on device” with 5GB free**

First run on-device threw:

*```
BNNS Graph Compile: failed to preallocate file … No space left on device
[Espresso::handle_ex_plan] exception=ANECF error: failed to load ANE model
```*

The literal error message says storage — but the device had 5GB free. What’s actually happening: CoreML’s `*.cpuAndNeuralEngine*` compute path routes through Apple’s ANE compiler service, which *lazily* compiles a device-specific graph on the **first prediction call**, not at model load. For a model this size, that compile step turned out to be unreliable on-device — a known class of flakiness, not a real storage problem.

Fix: switch to `.*cpuAndGPU*`, which uses Metal instead of the ANE compiler and sidesteps that path entirely, at a modest cost in inference speed.

*```swift
let config = MLModelConfiguration()
config.computeUnits = .cpuAndGPU
model = try IndicWav2VecHindi(configuration: config)
```*

**Bug 2: a crash inside the CTC decoder**

Second run got further — transcription actually started — and then crashed at:

*```swift
return decoder.decode(logits: output.logits)
```*

Here’s the interesting part. My Swift decoder reads the model’s output `*MLMultiArray*` as raw memory for speed:

*```swift
let pointer = logits.dataPointer.assumingMemoryBound(to: Float32.self)
```*

That assumes 4 bytes per element. I checked the actual on-device output spec:

*```python
spec = mlmodel.get_spec()
for out in spec.description.output:
 print(out.name, out.type)
# logits multiArrayType { dataType: FLOAT16 }
```*

There it was: even though I’d set `*compute_precision=FLOAT16*`, I hadn’t pinned the **output** dtype, so CoreML defaulted the `*logits*` output to float16 on-device — 2 bytes per element, not 4. My Swift code was reading twice as many bytes as the buffer actually contained, running off the end of allocated memory. My Python `*validate.py*` never caught this because `*mlmodel.predict()*` silently upcasts whatever it returns into a float32 numpy array — the exact same call that “validated” the conversion was hiding the bug.

The fix was one line back in the conversion script — force the output dtype explicitly:

*```python
outputs=[ct.TensorType(name=”logits”, dtype=np.float32)],
```*

Reconverted, reverified (`FLOAT32` confirmed via `*get_spec()*`, argmax agreement still 100%), recopied the `*.mlpackage*` into the app, and the crash was gone.

**Lesson**: if you’re converting a model for on-device use, don’t just validate with the conversion library’s own Python API — inspect the actual output tensor spec* (`mlmodel.get_spec()`)* and cross-check it against whatever your app-side code assumes about memory layout.

## The SwiftUI app

With a validated model and vocab in hand, the app itself is four small, single-purpose files:

- `**AudioRecorder.swift**` — taps `AVAudioEngine`’s input node, uses `**AVAudioConverter**` to resample the mic’s native format (usually 44.1/48kHz) down to the 16kHz mono float32 PCM the model expects.
- `**STTEngine.swift**`— loads the CoreML model (auto-generated as a Swift class named `IndicWav2VecHindi` by Xcode’s model codegen) and runs one forward pass over the fully-buffered utterance.
- `**CTCDecoder.swift**` — greedy CTC decoding: argmax per time step, collapse consecutive repeats, drop the blank token, map the word-delimiter token to a space.
- `**ContentView.swift**` — the record button and transcript view, wired to the above through a small `*ObservableObject*`.

One deliberate scope decision: this is **tap-to-record → tap-to-stop → transcribe**, not live streaming captions. wav2vec2 with a CTC head isn’t a causal/streaming architecture — running it incrementally on partial audio doesn’t produce anything meaningful. Batch-per-utterance is the honest MVP; real streaming would need a fundamentally different model or a chunked pseudo-streaming hack layered on top, which felt like the wrong thing to build before the basic pipeline even worked.

The whole thing was scaffolded with [XcodeGen](https://github.com/yonaskolb/XcodeGen) *(`project.yml` → `xcodegen generate`*) rather than a hand-written `*.xcodeproj*`, since `*pbxproj*` files are miserable to author by hand and XcodeGen’s YAML is trivial to read, diff, and regenerate.

## Adding text-to-speech, synced to a typewriter effect

As a follow-up, I added spoken playback of the transcript — with the text visually “typing out” in sync with the narration, rather than appearing all at once or the speech and text being unrelated.

The naive approach would be to estimate a typing speed from word count and animate against a timer. Instead, `AVSpeechSynthesizer` has a delegate callback built for exactly this:

*```swift
func speechSynthesizer(
 _ synthesizer: AVSpeechSynthesizer,
 willSpeakRangeOfSpeechString characterRange: NSRange,
 utterance: AVSpeechUtterance
) {
 guard let range = Range(characterRange, in: utterance.speechString) else { return }
 displayedTranscript = String(utterance.speechString[..<range.upperBound])
}
```*

This fires with the **actual** range of the string being spoken at that moment — driven by Apple’s own TTS engine, not a guess — so the revealed text and the narration stay aligned regardless of phrase length or speaking rate.

One more session-management detail: after recording, the audio session is configured for `.*record*` (input only), which can’t produce output. Speaking silently does nothing until you explicitly switch categories:

*```swift
try AVAudioSession.sharedInstance().setCategory(.playback, mode: .default)
try AVAudioSession.sharedInstance().setActive(true)
```*

Easy to miss, since there’s no error — it just plays no sound.

## Where it landed

- **Model**: `*ai4bharat/indicwav2vec-hindi*`, converted to a 602MB CoreML `.*mlpackage*`, float16 compute, flexible 1–30 second input, running on `.*cpuAndGPU*`.
- **Accuracy of conversion**: 100% argmax agreement between PyTorch and CoreML outputs.
- **App**: fully offline SwiftUI app — record, transcribe, hear it read back with synced text reveal — no network calls at any point after the model ships inside the app bundle.

The biggest practical takeaway from the whole exercise: conversion frameworks like `*coremltools*` are very good at getting a model **running** on-device, but their Python-side convenience APIs (auto-casting outputs, hiding on-device compute-unit behavior) can mask real bugs that only surface once Swift code touches the raw output directly. If you’re doing this kind of conversion yourself, budget time for on-device testing specifically — passing validation in Python is necessary, but it is not sufficient.

GitHub Link : [https://github.com/kabirdasbk07-debug/ai4bharat-stt-ios.git](https://github.com/kabirdasbk07-debug/ai4bharat-stt-ios.git)