Run an LLM in the Browser, No Backend, No API Key

Every AI feature you’ve built in React probably starts the same way: a fetch call to OpenAI, Anthropic, or your own inference server. That means an API key to manage, a per-token bill, network latency, and — if the feature touches anything sensitive — user data leaving the device.

use-browser-llm skips all of that. It's a headless React hook that runs a real LLM entirely inside the browser tab, using WebGPU. No server, no key, no data leaving the device.

How it actually runs a model client-side

Under the hood, use-browser-llm wraps [@mlc-ai/web-llm](https://github.com/mlc-ai/web-llm), which compiles and runs quantized LLMs directly on the GPU via WebGPU — the browser API that gives web pages near-native access to graphics hardware. Inference happens inside a dedicated Web Worker, so a generating model doesn't freeze your UI thread. Model weights get cached in IndexedDB, so the multi-gigabyte download only happens once; repeat visits load from cache.

The package itself ships no UI — no chat bubble, no spinner, no button. It’s deliberately headless, so it drops into whatever design system you’re already using, whether that’s Tailwind, Shadcn, CSS Modules, or plain CSS. You bring the components; the hook gives you the state machine and the functions that drive it.

Benefits at a glance

No API key to provision or rotate, and nothing to bill per token — inference runs on the user’s own GPU. No inference server to stand up, scale, or pay for, since there’s no backend in the loop at all. Genuine privacy for sensitive use cases, because prompts and responses never leave the browser tab. Works offline after the first model download, since weights are cached in IndexedDB. No network round-trip latency once the model is loaded — generation starts immediately instead of waiting on a request to a remote API. And it’s headless, so it costs you nothing in design flexibility: the hook exposes state and functions, you keep full control of the UI.

Requirements

• React 18+
• A browser with WebGPU support — currently Chrome and Edge, with coverage expanding. No WebGPU means no local inference, which the hook handles gracefully (more on that below).

Install and quickstart

npm install use-browser-llm

import { useState } from "react";
import { useBrowserLLM } from "use-browser-llm";

const MODEL_ID = "Llama-3.2-1B-Instruct-q4f16_1-MLC";

export function App() {
  const { status, progress, generate, isGenerating } = useBrowserLLM(MODEL_ID);
  const [reply, setReply] = useState("");

  if (status === "loading") {
    return <p>Loading model… {Math.round(progress * 100)}%</p>;
  }

  if (status === "error") {
    return <p>Something went wrong loading the model.</p>;
  }

  return (
    <div>
      <button
        disabled={status !== "ready" || isGenerating}
        onClick={async () => {
          const text = await generate([
            { role: "user", content: "Say hello in one short sentence." },
          ]);
          setReply(text);
        }}
      >
        {isGenerating ? "Generating…" : "Ask"}
      </button>
      <p>{reply}</p>
    </div>
  );
}

You pass a model id from @mlc-ai/web-llm's prebuilt model list — or undefined if you're not ready to load one yet, say, while the user is still picking a model from a dropdown. status walks through "idle" → "loading" → "ready" or "error". generate() only resolves once the model is "ready"; call it earlier and it rejects immediately rather than queuing silently.

A more complete example: chat UI with system prompt, streaming, and fallback

Putting the pieces together — a system prompt, streaming tokens, a cache-aware loading message, and a fallback for unsupported browsers — looks like this:

import { useState } from "react";
import { useBrowserLLM } from "use-browser-llm";

const MODEL_ID = "Llama-3.2-1B-Instruct-q4f16_1-MLC";

export function ChatPanel() {
  const { status, progress, cacheStatus, streamGenerate, abort, isGenerating } =
    useBrowserLLM(MODEL_ID);
  const [reply, setReply] = useState("");

  if (status === "unsupported") {
    return <p>This browser doesn't support local AI. Try the latest Chrome or Edge.</p>;
  }

  if (status === "loading") {
    return (
      <p>
        {cacheStatus === "downloading"
          ? `Downloading model… ${Math.round(progress * 100)}%`
          : "Loading model…"}
      </p>
    );
  }

  async function ask(question: string) {
    setReply("");
    for await (const token of streamGenerate([
      { role: "system", content: "You are a concise, friendly assistant." },
      { role: "user", content: question },
    ])) {
      setReply((prev) => prev + token);
    }
  }

  return (
    <div>
      <button disabled={status !== "ready" || isGenerating} onClick={() => ask("What's WebGPU?")}>
        {isGenerating ? "Generating…" : "Ask"}
      </button>
      <button disabled={!isGenerating} onClick={abort}>
        Stop
      </button>
      <p>{reply}</p>
    </div>
  );
}

This is a full, production-shaped flow: it never shows a blank screen on unsupported browsers, distinguishes a first-time multi-gigabyte download from an instant cached load, streams tokens as they arrive, and gives the user a way to cancel mid-generation.

Letting users switch models

Since modelId can be undefined, you can defer loading until the user picks one — handy for offering a smaller, faster model alongside a larger, more capable one:

const [modelId, setModelId] = useState<string | undefined>(undefined);
const { status, progress } = useBrowserLLM(modelId);

// modelId flips from undefined to a real id once the user selects from a dropdown;
// the hook starts loading automatically on the next render.

Streaming output, token by token

For a typical chat UI you don’t want to wait for the full response — you want tokens appearing as they’re generated. streamGenerate() returns an AsyncGenerator:

async function handleAsk() {
  setText("");
  for await (const token of streamGenerate([
    { role: "user", content: "Write a haiku about the ocean." },
  ])) {
    setText((prev) => prev + token);
  }
}

Breaking out of the loop early stops generation in the worker itself — you don’t need to separately call abort() unless you're stopping generation from somewhere outside the loop, like a "Stop" button elsewhere in the UI.

Cancellation and concurrency

generate() and streamGenerate() share one abort() that reaches into the worker and actually halts inference, not just the promise on the main thread. isGenerating stays true for the duration of either call, and the hook enforces one generation at a time — calling either method while one is already running rejects with HookBusyError instead of quietly queuing requests or corrupting output.

Handling browsers without WebGPU

This is the part most local-AI libraries get wrong: they either crash or hang when WebGPU isn’t available. use-browser-llm resolves status straight to "unsupported" when the browser has no usable WebGPU — no adapter, missing entirely, or only a software fallback — and never spins up a worker in that case:

if (status === "unsupported") {
  return (
    <p>
      Your browser doesn't support the local AI features on this page. Try the
      latest Chrome or Edge.
    </p>
  );
}

The error object in this state is an UnsupportedError with a .reason field ("no-navigator-gpu", "no-adapter", or "fallback-adapter"), so you can be specific in your messaging if you want to be.

Knowing whether the model is cached

First-time visitors are downloading gigabytes; returning visitors should load near-instantly. cacheStatus tells you which situation you're in, so you can show the right message:

const { cacheStatus } = useBrowserLLM("Llama-3.2-1B-Instruct-q4f16_1-MLC");
// "idle" | "checking" | "cached" | "downloading"

It’s informational rather than load-gating — check status for anything that actually needs to block on model readiness.

Errors, typed and exported

Four error types cover the failure modes: HookNotReadyError (called before "ready"), HookBusyError (called while already generating), UnsupportedError (no WebGPU), and WorkerCrashError (the worker died or hung mid-inference). All four are exported for instanceof checks, so error handling doesn't rely on parsing message strings.

The API surface, in full

function useBrowserLLM(modelId: string | undefined): {
  status: "idle" | "loading" | "ready" | "error" | "unsupported";
  progress: number;
  error: Error | null;
  cacheStatus: "idle" | "checking" | "cached" | "downloading";
  isGenerating: boolean;
  generationError: Error | null;
  generate(messages: ChatMessage[]): Promise<string>;
  streamGenerate(messages: ChatMessage[]): AsyncGenerator<string, void, void>;
  abort(): void;
};

ChatMessage is a plain { role, content } type defined by this package — not a re-export of @mlc-ai/web-llm's internal message shape — so consuming code never has to know the underlying engine exists.

Why this matters

Client-side inference isn’t a replacement for server-side LLMs in every case — you’re bound by the user’s hardware and by what quantized open models can currently do. But for the right use cases — offline-capable tools, privacy-sensitive features where data genuinely can’t leave the device, or just avoiding a per-token bill for a lightweight assistant feature — running inference in the tab itself is a real option now, not a demo gimmick.

npm install use-browser-llm

Links

npm: https://www.npmjs.com/package/use-browser-llm

GitHub: https://github.com/Muhammad-UmairAli/use-browser-llm