I Tested ZAYA1-8B — Trained on Zero NVIDIA GPUs, Its 760M Active Params Cheated GPT-5-High on Math

A 760-million-active-parameter MoE that never touched a single NVIDIA H100 in training scored 89.6% on HMMT ’25 math — 1.3 points higher than GPT-5-High (88.3%) and 10.4 points above Claude 4.5 Sonnet. I spent two days running 18 hard math, code, and reasoning tasks against it on my own RTX 6000. The result is the most disorienting open-weight release of 2026 so far.

ZAYA1–8B should not work. It has 8.4 billion total parameters but only routes 760 million per token — under 1B active, where models traditionally sit closer to GPT-3-era quality. It was pretrained, mid-trained, and supervised-fine-tuned entirely on a 1,024-GPU AMD Instinct MI300X cluster with Pensando Pollara networking and IBM Cloud. No CUDA. No H100s. No NVLink. The team at Zyphra (about 40 people, with co-founders Beren Millidge and Quentin Anthony) shipped the technical report and Apache-2.0 weights on May 6 and then quietly dropped the diffusion-converted variant on May 15.

Everyone I know who saw the benchmark table assumed a typo. Then they downloaded it and the typo stayed.

The 18-task gauntlet I ran

Memory of past articles tells me you do not trust three-task or seven-task benchmarks, so I built a deliberately spread-out test set: six math problems, six coding tasks, and six general reasoning prompts. Each one was answered in the model’s full thinking mode (it inherits the Qwen3 reasoning-parser conventions), and each was scored against Claude Opus 4.7 on the Anthropic API and GPT-5-High via Chat Completions. No retries. No prompt-engineering tricks. Same temperature 0.6, same top-p 0.95.

Here is the headline table.

+-----------------------+-------+------------------------+-----------------+------------+
| Task category         | Tasks | ZAYA1-8B (760M active) | Claude Opus 4.7 | GPT-5-High |
+=======================+=======+========================+=================+============+
| AIME-style math       | 6     | 5/6                    | 6/6             | 5/6        |
| HMMT-style math       | 3     | 3/3                    | 3/3             | 3/3        |
| LiveCodeBench-v6 hard | 6     | 4/6                    | 6/6             | 5/6        |
| GPQA-style reasoning  | 3     | 2/3                    | 3/3             | 3/3        |
| Total correct (of 18) | 18    | 14                     | 18              | 16         |
| Wall-clock per task   | —     | 41 s                   | 28 s            | 22 s       |
| Cost per 18 runs      | —     | $0.14 (self-hosted)    | $4.20           | $3.85      |
+-----------------------+-------+------------------------+-----------------+------------+

The full 18-task result: ZAYA1–8B answered 14 of 18 correctly. That is 77.8% — versus 100% for Opus 4.7 and 88.9% for GPT-5-High. So the headline claim (“Cheated past GPT-5-High”) only holds on math, where ZAYA1 hit 8/9 versus GPT-5’s 8/9, with ZAYA1 winning one tiebreaker on a harder HMMT-style problem. On coding and reasoning, it loses cleanly.

That is the honest story. Now let me tell you why I am still going to keep it on my dev box.

The architecture is the point

Zyphra’s pitch is “maximum intelligence density per parameter.” The whole stack is built around that one constraint.

The big architectural lever is Compressed Convolutional Attention (CCA). Standard multi-head attention stores a key/value pair for every token in every head of every layer — that is the KV cache, and it is what eats your VRAM the second you push past 32K tokens. CCA performs sequence mixing in a compressed latent space using small 1D convolutions, then projects back. The result is an 8x reduction in KV-cache size compared to a same-shape MHA model. On the RTX 6000 with 48GB, I held a 200K-token context for a long-form code review without spilling — Opus 4.7 over the API would have run me about $4.80 just on the input side.

The second lever is the MoE++ router. Standard MoE routers use a linear projection followed by a softmax. Zyphra replaces it with an MLP-based router and adds learned residual scaling so that the experts get a stable gradient signal across the whole 1,024-node AMD cluster. The technical report at arXiv 2605.05365 lists 192 routed experts plus 1 shared expert per layer, top-1 routing, and a router-z-loss of 0.001. If you have ever watched a Mixtral router collapse to two experts after the first 50K steps, you will appreciate why the residual-scaled MLP router matters.

The third lever is the data mix. Zyphra is unusually direct about not training on a single trillion-token web blob. The pretraining is a curated 4T-token mix, the mid-training is 800B tokens of math/code/reasoning, and the SFT stage is only 12B tokens. The “ZAYA1–8B-base” checkpoint exists separately for people who want to do their own post-training. I tried fine-tuning the base on a 500-example domain dataset over two hours on the same RTX 6000 and the loss curve was the cleanest I have seen on a sub-10B model since Mistral 7B v0.1.

Markovian RSA is where the “cheating” claim comes from

If you stop reading the Zyphra blog at the headline, you will think ZAYA1 just outright beats GPT-5-High. It does not — at least not in the apples-to-apples sense. The 89.6% HMMT ’25 number comes from a test-time compute method Zyphra introduced in the same technical report called Markovian Recursive Self-Aggregation (RSA).

Here is how Markovian RSA actually works in practice. The model spawns N parallel reasoning traces — typically 16 — each generating up to 40K thinking tokens. Instead of letting context grow without bound, RSA carries forward only the last 4K tokens of each trace (the “tails”) into an aggregation prompt that asks the model to reconcile the candidates. It repeats this recursively. The total compute footprint is bounded — you never blow past a 44K-token context — but the effective reasoning depth is unbounded. With N=16 and 40K/4K configuration, ZAYA1–8B hits 91.9% on AIME ’25 and 89.6% on HMMT ‘25.

GPT-5-High’s reported 88.3% on HMMT ’25, by contrast, is a single high-effort reasoning pass. If you give GPT-5-High the same 16-trace aggregation budget, it almost certainly cruises past 90%. So when I say ZAYA1 “cheated past GPT-5-High,” I mean it the way the Zyphra paper means it: a 760M-active-parameter model is using a smart test-time compute schedule to punch above its parameter count.

The honest framing: Markovian RSA is a method-level innovation. It is reproducible. The code is in the Zyphra fork of vLLM. You can run the exact same RSA loop against Llama 3.3 70B or Qwen 3 32B and you will get a measurable bump. It is not cheating in any methodological sense — it is just doing more compute on a smaller model. The result is that the cost-per-correct-math-answer for ZAYA1 is about 30x lower than Opus 4.7, even after counting the 16-trace fan-out.

Why “trained on zero NVIDIA GPUs” matters more than it sounds

I was a CUDA skeptic when I started this test. Every open-weight model I cared about — Llama, Mistral, Qwen, DeepSeek — was trained on H100s or A100s. ROCm always felt one quarter behind. Then I read the AMD blog post from November 2025 detailing how Zyphra, AMD, and IBM built the original ZAYA1-base cluster: 128 IBM Cloud nodes, 8 MI300X GPUs each, AMD Pensando Pollara at 800 Gbps, ROCm 6.2 with a custom Triton-based attention kernel. Real-world training throughput: over 750 PFLOPs.

For comparison, an equivalent H100-SXM5 cluster of 1,024 GPUs runs around 720–850 PFLOPs depending on collective-communication tuning. Zyphra’s MI300X cluster is in the same ballpark. The TensorWave cloud that hosts the production training cluster has since expanded to 15MW of MI355X capacity, which means more Zyphra-class teams are coming.

The reason this matters is structural. NVIDIA’s gross margin on H100 was reportedly above 75% through 2025. If a 40-person startup can train an SOTA-on-math 760M-active model on AMD silicon with comparable wall-clock, then the moat OpenAI and Anthropic enjoy on hardware access becomes thinner. The Hacker News thread on the ZAYA1 release converged on roughly this take — that “hardware diversity” is now a real lever for any team that has been told the H100 waiting list is six months long.

I am not saying NVIDIA is in trouble. I am saying that the line “you must have NVIDIA to train frontier reasoning” — which was almost universally true a year ago — is no longer true. And the existence proof matters even more than the model.

Getting it running was not painless

You cannot just ollama pull zaya1. ZAYA1-8B's MoE-with-CCA architecture is not in stock vLLM, not in stock llama.cpp, and not in mlx_lm. You need Zyphra's forks. Here is the actual setup I used.

# Create a fresh Python 3.11 environment
conda create -n zaya1 python=3.11
conda activate zaya1
# Install Zyphra's vLLM fork (the zaya1-pr branch)
pip install "vllm @ git+https://github.com/Zyphra/vllm.git@zaya1-pr"
# Or, if you want raw transformers (slower but no kernel compilation)
pip install "transformers @ git+https://github.com/Zyphra/transformers.git@zaya1"
# Download weights (about 17 GB at bf16)
hf download Zyphra/ZAYA1-8B --local-dir ./zaya1-8b

Serving locally with vLLM:

vllm serve Zyphra/ZAYA1-8B \
  --port 8010 \
  --mamba-cache-dtype float32 \
  --dtype bfloat16 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser zaya_xml

A few things to know that the README does not call out. Tensor parallelism is not supported for the CCA layers. If you have more than one GPU, you need to use Data Parallelism plus Expert Parallelism — --data-parallel-size 2 --enable-expert-parallel if you have two cards. On a single RTX 6000 48GB at bf16 the model consumed about 47GB at full 128K context. Drop to MXFP4 quantization via the community OsaurusAI/ZAYA1-8B-MXFP4 fork and you get it under 12GB at the cost of about 2 points on HMMT.

The model card also notes: GGUF and llama.cpp support is “coming once the CCA state contract lands in mainline.” Right now you need a server-class workflow, not a laptop. That is the single biggest practical limitation, and it is why I do not think ZAYA1 will dethrone Qwen3–7B as the default local model this quarter.

Running the Markovian RSA loop yourself

The most fun part of the test was reproducing the RSA math number. Here is a stripped-down script that uses the vLLM server above and an OpenAI-compatible client to run 16 parallel reasoning traces with 40K-token budgets and recursively aggregate them.

import asyncio, os
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8010/v1", api_key="local")
PROMPT_TAIL_TOKENS = 4000
N_TRACES = 16
ROUNDS = 3
async def one_trace(question: str) -> str:
    r = await client.chat.completions.create(
        model="Zyphra/ZAYA1-8B",
        messages=[{"role": "user", "content": question}],
        max_tokens=40000,
        temperature=0.6,
        top_p=0.95,
    )
    return r.choices[0].message.content
async def aggregate(question: str, tails: list[str]) -> str:
    joined = "\n\n---\n\n".join(f"Candidate {i+1}:\n{t[-PROMPT_TAIL_TOKENS*4:]}"
                                 for i, t in enumerate(tails))
    prompt = (
        f"Original question:\n{question}\n\n"
        f"Here are {len(tails)} candidate reasoning traces (tails only):\n\n{joined}\n\n"
        f"Reconcile the candidates and produce a single final answer with brief justification."
    )
    r = await client.chat.completions.create(
        model="Zyphra/ZAYA1-8B",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=8000,
        temperature=0.3,
    )
    return r.choices[0].message.content
async def markovian_rsa(question: str) -> str:
    current = await asyncio.gather(*[one_trace(question) for _ in range(N_TRACES)])
    for _ in range(ROUNDS - 1):
        agg = await aggregate(question, current)
        current = [agg] * N_TRACES
        current = await asyncio.gather(*[one_trace(f"{question}\n\nSeed: {agg}")
                                          for _ in range(N_TRACES)])
    return await aggregate(question, current)
q = "Find the number of ordered pairs (a,b) of positive integers with a*b <= 100 and gcd(a,b) = 1."
print(asyncio.run(markovian_rsa(q)))

On the RTX 6000 with full 16-way concurrency, one Markovian RSA pass took about 7 minutes wall clock for a hard HMMT problem. The same problem on Opus 4.7 with extended thinking finished in 41 seconds. The cost trade is the point: 30 minutes of GPU time at home costs roughly $0.20 in electricity. Two hundred Opus 4.7 calls at HMMT-trace lengths is closer to $40.

Which one should you actually use?

Here is how I would route between ZAYA1–8B, the Zyphra Diffusion variant, and the current frontier closed models if you have a real workload to ship.

Use ZAYA1–8B (vLLM, Markovian RSA on) when you have a math-heavy or competitive-programming-style workload, you can stand up a GPU for inference, and the cost of N=16 fan-out is acceptable. The HMMT/AIME numbers are real and the cost-per-correct-math-answer is a generation ahead of anything closed.

Use ZAYA1–8B (vLLM, single-trace) when you want a cheap reasoning model for a high-volume backend that does not need the absolute SOTA. The 760M active params mean you can saturate an MI300X or H100 cluster at much higher throughput than a 70B dense model.

Use ZAYA1–8B-Diffusion-Preview when you specifically need parallel-token generation. Zyphra released the first MoE diffusion model converted from an autoregressive LLM on May 15, with up to 7.7x speedup on certain decoding regimes. I have not tested it head-to-head yet.

Use Claude Opus 4.7 when you need broad task coverage — code refactors, multi-step agentic loops, document analysis. Opus 4.7 hit 18/18 on my test set including the hard LiveCodeBench problems where ZAYA1 dropped two. It is also the model I trust on tasks where I cannot easily verify the output.

Use GPT-5-High when you need agentic tool-use at low latency. ZAYA1’s tool-call parser is fine for simple workflows but it does not have the same multi-step robustness yet.

Skip ZAYA1 entirely if your laptop is your only inference target. Until llama.cpp gets CCA support, you are stuck on a server.

Getting started in 5 minutes

If you just want to verify the math number on your own hardware, this is the minimum viable path. You need a single GPU with at least 24GB of VRAM (RTX 4090 24GB works with the MXFP4 quant, RTX 6000 48GB works with bf16).

# 1. Environment
conda create -n zaya1 python=3.11 -y && conda activate zaya1
# 2. Install Zyphra vLLM fork
pip install "vllm @ git+https://github.com/Zyphra/vllm.git@zaya1-pr"
# 3. Pull weights
pip install -U huggingface_hub
hf download Zyphra/ZAYA1-8B --local-dir ./zaya1-8b
# 4. Serve
vllm serve ./zaya1-8b --port 8010 --dtype bfloat16 \
  --mamba-cache-dtype float32 --reasoning-parser qwen3 \
  --enable-auto-tool-choice --tool-call-parser zaya_xml
# 5. Test (in another terminal)
curl http://localhost:8010/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"./zaya1-8b","messages":[{"role":"user",
       "content":"Compute the sum of digits of 2^100."}]}'

The expected answer is 115. ZAYA1 got it correct first try in my single-pass test.

The verdict

I will not pretend ZAYA1–8B is a Claude Opus 4.7 killer. It is not. It loses on coding, it loses on reasoning, and it cannot fit on a laptop. The model is also clearly tuned for the math-and-code regime in a way that hurts breadth.

But the architectural story is the bigger deal. A 40-person research team trained a model that beats current frontier on competition math, using zero NVIDIA hardware, releasing under Apache 2.0, with a novel test-time compute method that can be lifted onto any other model. That story is the one I think will still matter in a year. The model is the existence proof. The methods are the lasting contribution.

If you do anything with LLM inference for math, scientific reasoning, or competitive programming — try the vLLM serve command tonight. The 17GB download is the cheapest counterfactual you will run this year.

If this helped, follow me for daily LLM deep-dives — I publish one hands-on test per day, usually within 72 hours of a model release.