No Server, No API Key, No Internet: An LLM in a Browser Tab

I built an offline recipe generator with Transformers.js. Here is what I learned, including the parts that went badly.

TL;DR

• Web AI means the model runs inside the browser instead of on a server. No API key, no bill per request, no data leaving the device.
• I built an offline recipe generator. You type “paneer, rice, half an onion” and a language model running in your tab writes a recipe. It works in airplane mode.
• The stack is Transformers.js running ONNX models on WebGPU, with a WASM fallback for older devices.
• The hard part is not the AI. It is the first download, because you are asking someone to pull 300MB to 1GB before they see anything useful.
• Small models are good now. They are also wrong with total confidence, slow on cheap phones, and completely dependent on the user’s hardware.
• Use this when privacy, offline support, or cost per request actually matter. Skip it if you just want a chatbot.

Live demo: Offline Recipe Generator. Load it once, turn off your WiFi, then try it again. That is the whole pitch.

I didn’t have a problem to solve

Let me be honest about the origin story. There wasn’t one. No moment in the kitchen, no user research, no gap in the market.

I read that Transformers.js could run a language model on WebGPU and I didn’t believe it. Not really. Not in a tab. Not on a phone. Not without some server quietly doing the actual work while the demo took the credit.

So the recipe app is an excuse. I picked it because it was the smallest honest test I could think of. It is a text task, the cost of a bad answer is one boring dinner, and sending “here is everything in my fridge” to a company’s server feels a bit silly.

What I wanted to know was whether the browser had grown a new ability while I was busy arguing about bundle size.

It had. Nothing leaves the device. Not the ingredients, not the prompt, not one network request. The model runs in the tab.

Words you need first

If you write React and you have been nodding politely through AI conversations, start here. Everyone else can scroll past.

Inference. Running a trained model to get an answer. Training is how the model is built. Inference is you asking it something. This article is only about inference.

Parameters. The numbers inside the model. A “0.5B” model has 500 million of them. More parameters usually means smarter and heavier.

Quantization. Squeezing those numbers from 32 bits down to 8 or 4 bits. A 1B model at full precision is around 4GB. At 4 bits it fits under 700MB. You lose a little quality and save a lot of download.

Tokens. Pieces of text the model reads and writes, roughly three quarters of a word each. Tokens per second is the speed number that matters.

ONNX. A file format for models that works outside Python. It is the bridge that lets a model trained in PyTorch run in your browser.

WebGPU. The browser API that gives JavaScript real access to the GPU. This is what makes local inference fast instead of just possible.

WASM. WebAssembly. Fast code running on the CPU. It is your fallback when WebGPU is missing. Slower, but it runs almost everywhere.

Transformers.js. Hugging Face’s JavaScript library for running these models in the browser. If you have used their Python library, pipeline() will feel familiar.

One thing that confuses people, and confused me: TensorFlow.js and Transformers.js are different tools. TF.js is a general ML runtime and it is great for things like pose detection or image classification. Transformers.js is built for transformer models in ONNX format, which is what you want once a language model is involved. I started with the first one in my head and ended up with the second.

What Web AI actually means

For a few years now, “adding AI” has meant the same shape every time. User types something, it goes to your server, your server calls someone else’s API, the answer comes back. You are a middleman with a latency problem and a monthly invoice.

Web AI changes the shape. The model becomes a static file. You download it once, cache it, and run it locally, the same way you cache a font.

Three things follow from that, and they are bigger than they sound.

Privacy stops being a promise. I don’t have a policy page saying I won’t store your ingredients. I have an app where storing them is impossible, because there is nowhere to send them.

Cost goes to zero. Not cheap. Zero. The user’s electricity is the only running cost. My app could get popular tomorrow and my bill would not move.

Offline actually works. Not the fake kind where you get a cached shell and a spinner. The app works on a plane. It works in a shop with one bar of signal. It works because there was never anything to reach.

How it works

The important part is about forty lines of code.

1. Load a model. One call handles the download, the caching, and the backend choice.

import { pipeline } from '@huggingface/transformers';

const generator = await pipeline(
  'text-generation',
  'onnx-community/Qwen2.5-0.5B-Instruct',
  { dtype: 'q4', device: 'webgpu' }
);

dtype: 'q4' picks the 4 bit version of the weights. device: 'webgpu' is the line that makes this usable. Remove it and you fall back to WASM on the CPU, which works but feels very different.

2. The weights land in Cache Storage. Transformers.js pulls the ONNX files from the Hugging Face CDN once. After that it reads from disk. This is why the second visit is instant and the first one is not.

3. The ingredients become a prompt. Small models follow clear instructions well and guess badly, so the system prompt does a lot of the work.

const messages = [
  {
    role: 'system',
    content:
      'You are a recipe generator. Reply with a title, ' +
      'an ingredients list, then numbered steps. ' +
      'Use only the ingredients given, plus salt, oil and water.',
  },
  { role: 'user', content: 'paneer, rice, half an onion' },
];

That last instruction, “use only the ingredients given”, removed more invented ingredients than any model upgrade did.

4. Tokens stream into the page. TextStreamer calls you back for each chunk, so text appears as it is generated.

import { TextStreamer } from '@huggingface/transformers';

const streamer = new TextStreamer(generator.tokenizer, {
  skip_prompt: true,
  callback_function: (text) => setRecipe((r) => r + text),
});

await generator(messages, {
  max_new_tokens: 512,
  temperature: 0.7,
  do_sample: true,
  streamer,
});

5. Run it in a Web Worker. This one is not optional. On the main thread, inference freezes everything. No scrolling, no cancel button, nothing. Move it to a worker and post the tokens back.

// worker.js
self.onmessage = async ({ data }) => {
  const streamer = new TextStreamer(generator.tokenizer, {
    skip_prompt: true,
    callback_function: (text) => self.postMessage({ type: 'token', text }),
  });
  await generator(data.messages, { max_new_tokens: 512, streamer });
  self.postMessage({ type: 'done' });
};

That is the whole app. No backend. Static files on a CDN.

The first load is the real problem

Here is the part that demo videos skip.

The first thing your user sees is a progress bar measured in hundreds of megabytes.

Everything I learned about web performance says this is unacceptable. We argue about 40KB bundles. We tree shake. And here I am asking someone to download a file the size of a movie before they can type “eggs”.

You cannot fix this. You can only design around it.

• Say the number out loud. “Downloading model, 340MB. One time only, then it works offline.” People forgive a big download. They do not forgive a surprise one.
• Default to the smallest model that works. Let curious users pick something bigger. Do not make the first run your heaviest option.
• Show real progress. Real percentage, real megabytes, no fake shimmer.
• Manage the cache. I delete the old model before downloading a new one, and there is always a visible “Clear cache” button. Quietly eating 3GB of someone’s phone is how an app gets deleted.

The progress hook is built in, so there is no excuse for a mystery spinner.

await pipeline('text-generation', modelId, {
  dtype: 'q4',
  device: 'webgpu',
  progress_callback: (p) => {
    if (p.status !== 'progress') return;
    setProgress({
      file: p.file,                      // which file
      percent: Math.round(p.progress),   // 0 to 100
      mb: (p.loaded / 1024 / 1024).toFixed(0),
    });
  },
});

And giving that storage back is one line, because it is a normal cache.

await caches.delete('transformers-cache');

Treat the download as onboarding, not as loading. It changes everything you build after it.

Where it breaks

I like this stack. I am also not going to pretend.

Small models are wrong with total confidence. A 360M model will tell you to simmer something for four hours, or add an ingredient you never mentioned. That is fine for recipes, where the worst case is a bad dinner. It is not fine for medical, legal, or money questions, and I treat that as a hard limit on what I build, not a disclaimer at the bottom of the page.

Hardware differences are huge. The same recipe takes two seconds on my desktop and around twenty five on a mid range Android running WASM. You cannot control this and you cannot test your way out of it. You can only check the device and set expectations before the user taps generate.

Browser support is uneven. WebGPU is solid in Chromium browsers and has been arriving elsewhere, but arriving is not the same as being on every phone your users own. Check current support yourself, keep the WASM path, and detect properly.

async function pickDevice() {
  if (!('gpu' in navigator)) return 'wasm';
  const adapter = await navigator.gpu.requestAdapter();
  return adapter ? 'webgpu' : 'wasm';
}

Checking 'gpu' in navigator is not enough on its own. The API can exist while requestAdapter() still returns null on a blocked driver, so ask for the adapter.

Memory limits are real. Mobile browsers kill tabs that use too much. A model that runs nicely on a laptop can crash a phone. This is why the model picker is not a nice extra. It is load bearing.

Context is small. These are not 200K token models. Long documents, long chat history, big retrieval setups: none of that fits here. I built a single shot tool instead of a chatbot on purpose, and a lot of the output quality comes from that one decision.

No cloud fallback, by choice. If your device cannot run the model, my answer is “not on this device”. It is not “let me send your data somewhere instead”. The moment you add that fallback, you have given up the only guarantee that made this interesting.

When this is the right call

A quick filter, because the hype is unhelpful in both directions.

Use on-device when the data is private by nature, the app needs to work offline, the task is narrow, or paying per request would kill the idea. Think journals, notes, personal finance, field tools, anything used in places with bad networks.

Use an API when you need strong reasoning, long context, tool use, or your users will leave rather than wait for a 300MB download. There is nothing wrong with this. A hosted model is often the correct answer.

The interesting apps sit in between. Local for the fast, private, boring 90 percent, and a server only when the user asks for more.

If you start tomorrow

Pick a task, not a chatbot. Narrow jobs make small models look clever. Open ended chat makes them look broken.

Write your prompt like an API contract. Be strict about the format. These models follow structure well and improvise badly, which is the opposite of what you are used to.

Test on the worst phone you own, early. Your laptop is lying to you about how this feels.

Build the download screen before you build the clever part. That screen is what most people will judge you on.

The strangest thing about this project is how ordinary it felt to build. No servers. No keys. No deploy pipeline beyond copying files to a CDN. Just a model file, a browser tab, and a GPU that was already sitting there doing nothing.

We spent years treating the browser as a thin client for someone else’s computer. That stopped being true a while ago. Most of us have not updated our mental model yet.

Try it with the WiFi off. That is when it stops feeling like a demo.

The recipe generator is live at LIVE_URL_HERE. The first load downloads the model, everything after that is offline. If you build something with Transformers.js, put it in the responses. I would like to see it.