Building a Private AI Translator with WebGPU and Transformers.js

How I built a fully private, local AI translator running directly in the browser

I wanted to build a translator that lives entirely in the browser — an app light enough to be shared as a simple post, but powerful enough to process LLMs locally on your own hardware.

Technology stack: Transformers.js v3 and ONNX format

For this experiment, the beating heart is Transformers.js

Transformers.js is designed to be functionally equivalent to Hugging Face’s transformers python library, meaning you can run the same pretrained models using a very similar API.

This library allows Hugging Face models to be mapped directly onto ONNX Runtime, leveraging WebGPU for hardware acceleration.

The Open Neural Network Exchange (ONNX) is an open standard format created to represent machine learning models. Supported by a robust community of partners, ONNX defines a common set of operators and a common file format to enable AI developers to use models with a variety of frameworks, tools, runtimes, and compilers.

However, keep in mind:

• Download: You’ll need to download a few GB of data (depending on the model version) on first run.
• Hardware: You need a GPU and a browser that supports WebGPU (updated Chrome or Edge).
• Library: We’ll use Transformers.js (v3), which enables Hugging Face models to run on WebGPU.

How does the code work?

import { pipeline, env } from '@huggingface/transformers';

async function initEngine(modelPath) {
    try {
        // WebGPU: 8-bit 
        return await pipeline('translation', modelPath, {
            device: 'webgpu',
            dtype: 'q8', 
            progress_callback: (p) => console.log(`Loading: ${p.progress}%`)
        });
    } catch (e) {
        console.warn("WebGPU failed, using WASM (CPU):", e);
        return await pipeline('translation', modelPath, { device: 'wasm' });
    }
}

• Quantization: This reduces model precision but significantly lowers RAM usage.
• ONNX model: Transformers.js doesn’t use the original PyTorch files, but a version converted to ONNX (optimized for execution on different hardware).
• Pipeline: pipeline('translation', ...) handles everything: downloading, tokenization, and generation.

Model Avalability

My idea was running Google’s TranslateGemma-4b-it directly in the browser. A 4-billion parameter model running locally.

1. Model Availability: Many cutting-edge models aren’t “browser-ready.” They require conversion to the ONNX format.
2. VRAM Constraints: 4B models, even when quantized, are heavy. For many consumer-grade laptops, allocating 2GB+ of VRAM within a browser sandbox leads to immediate crashes or the dreaded “Out of Memory” errors.

Models can be exported to ONNX format using Hugging Face’s optimum :

from optimum.onnxruntime import ORTModelForSeq2SeqLM
from transformers import AutoTokenizer

# Il modello scelto: Leggero, specifico e performante
model_id = "Helsinki-NLP/opus-mt-en-it"

# Caricamento e conversione automatica in ONNX
model = ORTModelForSeq2SeqLM.from_pretrained(
    model_id,
    from_transformers=True
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Salvataggio locale dei file per Transformers.js
model.save_pretrained("./onnx-marian-en-it")
tokenizer.save_pretrained("./onnx-marian-en-it")

Optimizing the Selection Strategy (Model Routing)

During testing, I didn’t use a single model for everything. For common language pairs (EN/FR/DE), small models like T5 are unbeatable in speed — but they fail miserably with Italian (often mistaking it for German).

function getOptimalModel(src, tgt) {
    const coreLangs = ['en', 'fr', 'de'];
    // T5-Base (fast)
    if (coreLangs.includes(src) && coreLangs.includes(tgt)) {
        return 'Xenova/t5-base';
    }
    // over 200 languages
    return 'Xenova/nllb-200-distilled-600M';
}

At the end of my tests, I settled on OPUS-MT models from the Helsinki-NLP group. These are extremely lightweight and optimized for specific language pairs, such as English–Italian and others.

On Hugging Face, you can find models already available in ONNX format for many language pairs:

https://huggingface.co/models?search=xenova/opus

There’s also a Helsinki-NLP Space that uses their models directly via API.

But for our test, we’ll use a Gemini-generated app (via prompt) that relies on ONNX models.

Loading ONNX Model based on language pairs

Run ONNX Model

Give it a try

The full code and conversion scripts are available in my repository. Clone it, run a local server, and experience the “vibe” of local neural translation.

the application is here: https://mwzero.github.io/software-as-a-post/translator-onnx/

Post-Mortem Notes

I sent the link to a few friends to try it on their smartphones. The results? a disastrous.

Most experienced cryptic errors, and in some cases, the mobile browser simply crashed or forced a system restart.aaaaa

While WebGPU is making waves on Desktop, mobile support is still a fragmented mess. If the browser doesn’t support the specific GPU instructions, it tries to fallback to WASM (CPU), which turns the phone into a pocket-sized heater before the OS kills the process to save itself.

Conclusions

WebGPU promises miracles, but local hardware reality is different. In my tests:

• Loading latency: Models like NLLB-200 (600M+ parameters) require downloading hundreds of MB into the browser’s IndexedDB cache.
• Inference speed: Even with dtype: 'q8' (8-bit quantization), more accurate models were very slow on my machine. If a cloud API responds in ~100 ms, a local model may take 2–5 seconds for a single complex sentence.
• Memory errors: I encountered GPU context crashes (error 77285704) when trying to load dense models on integrated GPUs with limited VRAM.
• Specialized models: small models delivered more than satisfactory results.

The architecture for running AI in the browser exists — and it’s powerful:

Strengths

• Absolute privacy
• Zero inference cost for developers
• Easy distribution (Software as a Post)

Weaknesses

• ONNX models are not always available
• Performance degrades rapidly as model size grows

We need even smaller and more optimized models to make the experience truly smooth.

Links

• transformer.js: https://huggingface.co/docs/transformers.js/index and examples: https://github.com/huggingface/transformers.js-examples
• ONNX Runtime: https://onnxruntime.ai/
• a curated collection of pre-trained, state-of-the-art models in the ONNX format: https://huggingface.co/onnxmodelzoo/models
• Transformer.js Github repo: https://github.com/xenova
• https://huggingface.co/models?search=xenova/opus
• https://github.com/mwzero/software-as-a-post/tree/main/translator-onnx

Let’s Connect!

My Name is Maurizio Farina. If you have any questions or feedback, feel free to reach out to me.

Follow me on Medium | X | Linkedin | GitHub

If you like reading or sharing MVP stories through vibe coding take a look at the ‘Software as a Post’ channel.

[embed]Software as a Post Software as a Post: Build Once. Share Everywhere. Measure Everything.medium.com