The On-Device AI Showdown: Core AI vs. LiteRT-LM

What Apple’s WWDC 2026 framework switch and Google’s LiteRT-LM stack really mean once you stop reading the keynote slides and start reading the architecture.

Every mobile engineer has had this moment. You wire up a feature against a cloud LLM API, it works beautifully in the demo, and then someone pulls up the projected token bill at scale and the room goes quiet. Multiply a few thousand tokens per request by a few million daily users and “just call the API” stops being an engineering decision and becomes a unit-economics problem.

That tension is why both Apple and Google spent their 2026 developer conferences talking less about the cloud and more about the silicon already in your users’ pockets. At WWDC on June 8, Apple announced Core AI, retiring nine-year-old Core ML as its primary on-device intelligence framework. Google has been steadily shipping LiteRT-LM — the open-source orchestration layer already powering on-device GenAI in Chrome, ChromeOS, and the Pixel Watch — as the runtime for Android’s generative future.

Two of the most powerful companies on earth are making the same bet: that a meaningful slice of LLM inference is about to migrate off the server and onto the device. Here’s how each is making it, where the architectures diverge, and what it means if you ship software for a living.

Why on-device LLMs matter

The case rests on four pillars, and only one is cost.

Token economics. A cloud call has a marginal cost that never hits zero. On-device inference has a brutal up-front cost and then a marginal cost of essentially nothing. For high-frequency, low-stakes work — summarizing a notification, rewriting a sentence — a server round-trip per invocation is economically absurd once the hardware can do it for free.

Privacy. “Data never leaves the device” is a different threat model entirely: no transit, no server logs, far less regulatory surface. For health, finance, and messaging apps, that’s often the difference between shipping a feature and killing it in legal review.

Latency. No network in the path means time-to-first-token is bounded by your compute, not someone’s congested LTE connection. For autocomplete, inline rewriting, and voice, that responsiveness is the product.

Offline. Planes, subways, coverage gaps, and the developing world’s intermittent connectivity. A feature that needs five bars isn’t a feature.

The counterweight is the battery and thermal tax. Sustained decode lights up the GPU or NPU, and phones are thermally constrained slabs of glass. This is exactly why the smart architecture is hybrid: run the cheap, frequent, privacy-sensitive work on-device and escalate the rare, heavy work to the cloud. Both Apple and Google landed on that posture, which tells you it isn’t a fad.

If 2023–2024 was the cloud-LLM land grab, 2025–2026 is the year on-device inference became a platform primitive.

Apple’s Core AI

Core ML was conceived in 2017 for the ML of 2017 — image classifiers, regressors, tree ensembles. It was never built for autoregressive token streaming, multi-gigabyte weights, KV-cache management, or agent-style tool calling.

Core AI, announced at WWDC 2026 alongside iOS 27, closes that gap. The rename from “ML” to “AI” is deliberate — Apple is reframing the whole developer surface around generative and agentic workloads. The headline shifts, based on the keynote and Bloomberg’s reporting:

• Native async, streaming inference as a first-class concept, not a bolt-on.
• Large-model memory handling built into the framework’s assumptions.
• Third-party model integration — bring a fine-tuned Llama or Mistral, instead of being locked to .mlmodel.
• A unified Foundation Models surface — Apple’s own on-device model, third-party models, and custom deployments under one dispatch layer, on-device by default, cloud delegation gated behind explicit user permission.
• Model Context Protocol (MCP) support, which is what turns “AI that answers” into “AI that acts” inside your app.

The underlying pitch is the one Apple always sells: tight hardware/software integration targeting the Neural Engine and unified memory. Apple’s existing on-device foundation model already runs at roughly 30 tokens/sec on an iPhone 15 Pro at zero API cost — a useful baseline for what “good enough” feels like here.

A candid caveat: Core AI is one week old as of writing. The exact API surface and the PyTorch-to-Core-AI conversion path are still settling. Treat this as representative of the announced shape, not a copy-paste contract:

// Illustrative — iOS 26's Foundation Models pattern, which Core AI extends.
// Verify against the iOS 27 SDK before shipping.
import FoundationModels
let session = LanguageModelSession(
    instructions: "You are a concise on-device assistant."
)
let stream = session.streamResponse(to: "Summarize: \(emailBody)")
for try await partial in stream {
    updateUI(with: partial.content)   // token-by-token, on the Neural Engine
}

You don’t think about delegates, kernels, or which accelerator you’re on — the OS decides. The trade-off: you’re entirely inside Apple’s ecosystem, on Apple’s timeline.

Google’s ML Kit GenAI + LiteRT-LM

Google’s approach is the philosophical opposite. Where Apple gives you one opaque, vertically-integrated stack, Google gives you a layered, open pipeline you can enter at any altitude:

• LiteRT (formerly TensorFlow Lite) — the runtime that executes a single model on CPU, GPU, or NPU.
• LiteRT-LM — the open-source C++ orchestration layer where the hard LLM machinery lives: KV-cache management, session cloning, prompt caching, stateful multi-turn inference. Already in production in Chrome, ChromeOS, and Pixel Watch.
• The LLM Inference API — high-level Kotlin/Swift/JS/Flutter bindings most developers actually touch.
• ML Kit GenAI — batteries-included APIs for summarization, rewriting, and image description that hide the layers below.

That layering is the point: a startup that wants summarization calls ML Kit; a team needing custom KV-cache behavior drops to LiteRT-LM.

The harder problem Google is solving is that Android isn’t one device — it’s a fragmented universe of Snapdragon, Dimensity, and Tensor silicon. LiteRT-LM absorbs that through backend delegation, surfacing as ML Drift (a GPU path Google claims is up to 25× faster than CPU for Transformers), unified NPU access across Qualcomm/MediaTek/Google Tensor, aggressive 4-bit/8-bit quantization via the AI Edge Quantizer, and Multi-Token Prediction drafters (introduced with Gemma 4, up to 3× faster — speculative decoding on a phone). The model story is open too: tuned for Gemma 4 (incl. the 12B variant) but supporting Llama, Phi, and custom PyTorch models.

The contrast is the whole story: Apple optimizes a narrow surface to perfection; Google builds an open layer that has to survive contact with chaos.

Deploying a PyTorch model, side by side

iOS — PyTorch → conversion → Core AI → Neural Engine. The classical path uses coremltools to convert a traced model to .mlpackage with quantization applied during conversion. Core AI keeps a conversion-based workflow but widens what's accepted; the LLM-specific path is still being documented.

import coremltools as ct, torch
traced = torch.jit.trace(model, example_input)
mlmodel = ct.convert(traced, convert_to="mlprogram",
                     compute_units=ct.ComputeUnit.ALL)  # CPU+GPU+ANE
mlmodel = ct.optimize.coreml.palettize_weights(mlmodel, nbits=4)
mlmodel.save("Model.mlpackage")

Android — PyTorch → LiteRT-LM → LLM Inference API → Snapdragon NPU. Conversion runs through AI Edge Torch directly to the LiteRT format.

import ai_edge_torch, torch
edge_model = ai_edge_torch.convert(model.eval(), (torch.randn(1, 128),))
edge_model.export("model.tflite")  # then quantize 4-bit/8-bit

val options = LlmInference.LlmInferenceOptions.builder()
    .setModelPath("/data/local/tmp/gemma4-e2b-int4.litertlm")
    .setMaxTokens(1024)
    .setPreferredBackend(LlmInference.Backend.GPU)  // or NPU
    .build()
val llm = LlmInference.createFromOptions(context, options)
llm.generateResponseAsync(prompt) { partial, done -> appendToUi(partial) }

iOS gives you mature tooling and excellent Neural Engine integration inside a closed pipeline. Android gives you direct PyTorch conversion, explicit backend selection, and real model portability — and hands you the fragmentation tax in return.

Architecture, head to head

Hardware acceleration — Apple: Neural Engine + GPU via unified memory, OS-managed. Google: CPU / GPU (ML Drift) / NPU across three vendors, via unified delegation.

PyTorch support — Apple: via conversion, LLM path still emerging. Google: direct via AI Edge Torch.

Quantization — Apple: palettization in the toolchain. Google: AI Edge Quantizer 4/8-bit, plus LoRA fine-tuning.

Developer experience — Apple: highly abstracted, the OS decides everything. Google: layered, you choose your altitude.

Openness & portability — Apple: closed, historically locked formats, opening up with Core AI. Google: open source, multi-platform, broad model support.

Memory efficiency — Apple: excellent on uniform Apple Silicon. Google: strong, but varies across fragmented SoCs.

Both default to offline, on-device-first inference. The split is philosophical: Apple’s columns are about consistency, Google’s about reach. Neither is better in the abstract — they optimize different definitions of the word.

What the benchmarks actually show

I haven’t personally benchmarked these devices, so I won’t pretend I did — the numbers below are published figures (Beebom’s April 2026 cross-device test, Google’s Developer Blog, Qualcomm), each sourced inline. Google’s AI Edge Gallery app ships an in-app benchmark if you want to reproduce them on your own hardware.

The one thing to internalize: prefill and decode are different worlds. Prefill processes your whole prompt before the first token (compute-bound, where NPUs shine, determines time-to-first-token). Decode is the token-by-token rate after that (memory-bandwidth-bound, determines whether streaming feels fast). A number that doesn’t say which it’s measuring is marketing. For reference, human reading speed is ~4–7 tok/s, so anything above ~20 tok/s decode outruns the reader.

Published figures for Gemma 4-class models on current flagship silicon:

• Gemma 4 E2B, Galaxy S26 Ultra (Snapdragon 8 Elite Gen 5), GPU: ~48.6 tok/s decode (Beebom, Apr 2026)
• Gemma 4 E2B, iPhone Air (A19 Pro), GPU: ~51.3 tok/s decode (Beebom, Apr 2026)
• Gemma 4 E2B, Dragonwing IQ8, NPU: ~3,700 tok/s prefill / ~31 tok/s decode (Google Dev Blog)
• FastVLM vision, Snapdragon 8 Elite Gen 5, NPU: ~11,000 tok/s prefill, 0.12s TTFT on a 1024×1024 image (Google Dev Blog)

Three observations fall out:

The NPU prefill numbers are the real headline. Thousands of tok/s of prefill means you can stuff a document into context and still see a first token in well under a second — the difference between a feature that feels instant and one that feels broken.

Decode scales down hard with model size. ~4B-class (Gemma 4 E4B) is comfortably interactive on a flagship, well above reading speed — the sweet spot for shipping today. ~8B-class is the honest middle ground: published mobile figures for 4-bit 8B models land in the low-to-mid teens of tok/s, fine for “tap and wait a beat,” marginal for live streaming. ~12B (Gemma 4 12B) is supported but firmly heavy — memory and prefill dominate, decode is slow, and you should expect thermal throttling within a couple of minutes of hard use.

Thermal is the asterisk on every number. Benchmarks are short bursts. A real multi-minute conversation pushes the SoC into throttling and your 48 tok/s quietly becomes 30. None of the burst benchmarks capture this.

The product takeaway: size your model to the interaction, not the leaderboard. A 2–4B model that responds instantly and never throttles beats a 12B model that’s marginally smarter but stutters, for almost every real mobile feature.

Who has the better architecture?

Resist crowning a winner — they optimize for different things.

Apple wins on predictability. Owning the whole stack means ruthlessly efficient Neural Engine integration and a DX that hides nearly every knob, with consistent behavior across a small, well-characterized device set. The cost: you live on Apple’s terms, in Apple’s ecosystem, on Apple’s cadence — and Core AI is new enough that some polish is still a promise.

Google wins on reach. Broad model support, genuine cross-platform portability, an open-source runtime you can inspect, and a faster experimentation loop. The cost: fragmentation — the same code that flies on a Snapdragon 8 Elite limps on budget silicon, permanently your problem to test.

If you build only for iPhone and value effortless integration, Apple’s model is hard to beat. If you need the full Android install base, want model freedom, or won’t be locked to one roadmap, Google’s openness is the entire point. “Better” is just a function of which constraints you signed up for.

The bigger picture

Two of the most important computing platforms on earth now treat on-device inference as a default, not an experiment. The immediate consequence: expensive server-side inference becomes optional for a huge class of use cases. Summarization, rewriting, classification, extraction, simple agentic tasks — the bread and butter of most “AI features” — increasingly need a recent phone and a 2–4B model, not a data center. For startups, that quietly rewrites the unit economics of what’s fundable.

It also opens the door to on-device agents. With MCP arriving in both stacks, the local model isn’t just answering — it’s calling tools and acting on app data without a round-trip, under the device’s privacy boundary. A personal agent that acts on your data without it ever leaving your phone is a new product category, not a faster old one. And it pushes privacy-first AI from a compliance checkbox toward the path of least resistance.

The throughline: the smartphone is being recast as a personal AI computer — running frontier-ish intelligence locally, persistently, and privately, escalating to the cloud only when it genuinely has to.

The AI platform war has been fought in data centers for three years, measured in GPU clusters and pricing tiers. WWDC and I/O 2026 are where that started to change. Apple retiring Core ML for Core AI and Google maturing LiteRT-LM into a production runtime aren’t framework housekeeping — they’re both companies declaring that the next layer of intelligence belongs on the device.

The next AI platform war may not be fought in the cloud — it may be fought inside the phones already sitting in our pockets.

Figures cited from published sources (Beebom, Google Developers Blog, Qualcomm) as of June 2026. Core AI was announced at WWDC on June 8, 2026; its detailed API is still being documented — verify Apple-specific snippets against the shipping iOS 27 SDK.