Local LLM on iPhone: which runtime is actually fastest? I benchmarked MLX, llama.cpp, LiteRT-LM, and CoreML

Four on-device runtimes, same models, same prompt, same harness — on a real iPhone 17 Pro.

You want to run a local LLM on iPhone.

But there are several runtime options, and it’s genuinely not obvious which to pick.

And clear head-to-head benchmarks are hard to find. The contenders:

• MLX — Apple charging into the on-device LLM scene, all-in.
• llama.cpp — the name in local LLM: mature, well-worn, huge community.
• LiteRT-LM — Gemma-4-only, but Google’s purpose-built super-soldier.
• CoreML-LLM — runs on the Apple Neural Engine, the unit that’s been left out in this GPU/Metal era. I built this one… and, ah, can it even compete?

So — fine, let’s just do it.

On iPhone 17 Pro (A19 Pro), I ran the same models through all four on-device runtimes and measured decode speed and memory.

The verdict:

“For local LLM on iPhone, default to MLX.”

“For Gemma 4 specifically, LiteRT-LM is king.”

Conclusions first

• Decode speed: Qwen 3.5 2B → MLX is fastest (61 tok/s). Gemma 4 E2B → LiteRT-LM wins decisively (55 tok/s).
• Memory: CoreML / ANE (Apple Neural Engine) wins big — Qwen 3.5 2B in just 241 MB (~1/5 of MLX). Slowest decode, but hey: nice work, CoreML.
• Use-case picks at the end.

Setup

• Device: iPhone 17 Pro (A19 Pro, iOS 26.4.2)
• Runtimes: MLX Swift / llama.cpp / LiteRT-LM / CoreML (ANE)
• Models: Gemma 4 E2B, Qwen 3.5 2B (4-bit class)
• Task: short-chat (128-token response)
• Aggregation: median of 3 cold runs
• Metrics: decode tok/s (higher = better), peak memory MB (lower = better)

Result 1 — Decode throughput (tok/s, higher is better)

• Gemma 4 E2B: 🔴 LiteRT-LM 55.4 🏆 · 🟣 MLX 47.5 · 🔵 llama.cpp 37.8 · 🟠 CoreML/ANE 33.4
• Qwen 3.5 2B: 🟣 MLX 61.2 🏆 · 🔵 llama.cpp 39.1 · 🟠 CoreML/ANE 27.9 · 🔴 LiteRT-LM — (Gemma-only)

On Gemma 4 E2B, LiteRT-LM wins by a clear margin. LiteRT-LM is Google’s on-device inference runtime, running Gemma in its native .litertlm (INT4-QAT) format on the Metal GPU. Their model × their runtime × their format — the optimization compounds. The Swift API was under construction for the longest time; whoever was on it, this is a strong result.

On Qwen 3.5 2B, MLX-Swift takes it at 61 tok/s. Apple’s on-device LLM push is the real deal. (LiteRT-LM doesn’t enter this row — its catalog is Gemma-only.)

Result 2 — Peak memory (MB, lower is better)

• Gemma 4 E2B: 🔴 LiteRT-LM 641 🏆 · 🟠 CoreML/ANE 1,187 · 🟣 MLX 2,900 · 🔵 llama.cpp 3,156
• Qwen 3.5 2B: 🟠 CoreML/ANE 241 🏆 · 🟣 MLX 1,279 · 🔵 llama.cpp 1,479

CoreML / ANE wins this, hard. Qwen 3.5 2B in 241 MB — about 1/5 of MLX (1,279) and llama.cpp (1,479). This is the chunked-MLKV path (Qwen35MLKVGenerator in CoreML-LLM): chunked weights + KV cache streamed through the Neural Engine.

If your binding constraint is RAM — running a 2B-class model on memory-tight iPhones, or letting your LLM coexist with the rest of your app without fighting over memory — the ANE is a very strong option.

Fairness notes

• CoreML / ANE is engineered for memory and power efficiency over raw throughput. First-load triggers ANE compilation, so load times are longer. The decode count is the number of streamed pieces (≈ tokens).
• LiteRT-LM has no per-call output cap, so it runs to EOS (a full ~458-token reply); the others stop at 128. Decode tok/s is a steady-state rate, so the comparison still holds. Numbers come straight from LiteRT-LM’s own benchmark counters (getBenchmarkInfo).
• All models are 4-bit class, but the quantization schemes differ slightly: MLX 4-bit / GGUF Q4_K_M / LiteRT INT4-QAT / CoreML INT4-palettized (Gemma) and INT8 (Qwen).

Which runtime should you use?

• Raw speed + a broad model catalog → MLX-Swift. Fastest on Qwen, dead-simple from Swift, and mlx-community has the models. The default first pick for local LLM on Apple devices.
• You’re on Gemma → LiteRT-LM. Best on both speed and memory for the Gemma family. Can’t beat it on its home turf.
• Memory-constrained / multi-purpose apps → CoreML / ANE.241 MB for a 2B-class model is wild. If your throughput budget is “human reading speed,” you get the memory and power efficiency for free.
• Portability across non-Apple platforms → llama.cpp. GGUF ecosystem, runs everywhere. No flashy peaks, but a dependable workhorse.

How I measured it (and how you can reproduce)

Every run is driven from a Mac via devicectl in fully-headless mode — nothing tapped on the phone. Models are side-loaded from Mac to device. The raw JSONL files and the chart all live in the repo. A pull request is one benchmark row: one runtime × one model × one device.

👉 **github.com/john-rocky/apple-silicon-llm-bench**

The engineering side — automating iPhone benchmarks end-to-end, the git-LFS skirmish, SwiftPM’s unsafe-flags rule, *@preconcurrency*imports — is a story of its own. Separate post.

Wrap-up

iPhone local LLM, in one line: speed → MLX or LiteRT-LM; memory → CoreML / ANE.

Hope it helps with your own on-device LLM work! 🐥

Work & technical consulting

I provide development support across iOS / Android / LLM / CoreML / LiteRT / ARKit / LiDAR / camera & image recognition.

I’m especially good at the parts that get stuck on real devices and in production:

• On-device AI inference, CoreML conversion, and speedups on mobile
• Recognition apps combining Camera / Vision / YOLO / OCR / CLIP
• 3D coordinate estimation, measurement, and on-site scanning with ARKit / LiDAR
• Taking over PoCs, performance work, and pre-delivery technical rescue
• Rethinking “works in the demo, flaky in the field” AI apps

Beyond proof-of-concept, I help land things in an actually-usable form — factoring in real-world constraints, UX, and operations.

If you’re stuck on something similar, reach out at rockyshikoku@gmail.com, or via X / LinkedIn.

X ・ Medium ・ GitHub