Bleeding Edge or Bleeding Out? The Quest for vLLM on NVIDIA Blackwell

Why my 57GB BF16 experiment failed, and what it reveals about the gap between hardware delivery and software reality.

A technical investigation into why cutting-edge inference frameworks struggle with bleeding-edge hardware — and what it teaches us about the GPU software stack

The Hypothesis

After discovering that vLLM’s FP8 CUTLASS kernels couldn’t compile for NVIDIA’s GB10 (Blackwell) architecture, I had a theory: maybe the problem wasn’t vLLM itself, but specifically the FP8 quantization path. After all, Blackwell’s SM_121 compute capability is brand new, and quantization kernels are notoriously architecture-specific.

What if we could bypass this limitation entirely by testing with BF16 (bfloat16) precision — no quantization, no exotic data types, just pure floating-point inference?

The model of choice: Qwen3–30B-A3B-Instruct-2507, a 30-billion parameter Mixture-of-Experts (MoE) model. At 57GB in BF16, it would fit comfortably in our GB10’s 120GB unified memory. The stage was set for what I hoped would be a success story.

Spoiler: it wasn’t. But the failure turned out to be far more interesting than any success could have been.

The Download: Watching Paint Dry at Gigabytes Per Minute

At 21:20 UTC, I launched the vLLM server. The first few log lines were promising:

INFO: Using FLASH_ATTN attention backend INFO: Using TRITON backend for Unquantized MoE

FlashAttention! That was good news. FlashAttention is a modern, Blackwell-compatible kernel that avoids Triton’s compilation bottlenecks. But that second line? “TRITON backend for Unquantized MoE” — I made a mental note of it, though I didn’t realize yet how critical it would become.

Then came the waiting game. HuggingFace’s servers delivered the 57GB model at a steady 3–4 GB per minute. I wrote a monitoring script to check progress every 30 seconds:

Bash

Check 1/20–21:23:37 | Model cache size: 18G Check 4/20–21:25:07 | Model cache size: 28G Check 9/20–21:27:38 | Model cache size: 44G

There’s something meditative about watching large models download. Each checkpoint shard represents months of training, billions of tokens processed, uncounted GPU-hours. By 21:29:32, all 57GB had landed. 539.532000 seconds, the logs reported, with characteristic machine precision.

The Loading Phase: Sixteen Safetensors and a Prayer

Model loading began immediately. The logs showed a progress bar:

Loading safetensors checkpoint shards: 6% Completed | 1/16 [00:21<05:16, 21.11s/it]

Sixteen shards, about 21 seconds each. During this phase, I monitored GPU memory with nvidia-smi. A peculiarity of the GB10: its unified memory architecture makes traditional GPU memory queries return [N/A] for most fields. But the process list told the real story: 58606 MiB allocated to the EngineCore process. The model was loading into memory successfully.

At 21:32:38, we hit 56% complete (9/16 shards). At 21:33:08, 62% (10/16 shards). My monitoring script hit its 20-check limit and exited. The server was still running, still loading shards. I waited.

The Crash: When PTX Met SM_121

At 21:35:06, everything fell apart.

torch._inductor.exc.InductorError: RuntimeError: Failed to run autotuning code block: No valid triton configs. PTXASError: PTXAS error: Internal Triton PTX codegen error ptxas stderr: ptxas fatal : Value ‘sm_121a’ is not defined for option ‘gpu-name’

Visualizing vLLM Stack

The PTX assembler didn’t recognize SM_121.

Let that sink in. We’re running PyTorch 2.9.1 with CUDA 13.0, on NVIDIA’s latest Blackwell architecture, and the toolchain’s own assembler is rejecting the GPU’s architecture identifier. This wasn’t a model compatibility issue. This wasn’t a memory problem. This was a fundamental gap in the software stack.

Think of PTX as the assembly language for NVIDIA GPUs. Triton (the compiler) wrote a letter addressed to “The Blackwell Architecture,” but ptxas (the mail carrier) looked at the address, said “This place doesn’t exist,” and threw the letter in the trash.

The Technical Deep Dive: Why MoE Models Failed

Here’s what I learned by tracing through the error. vLLM’s backend selection is architecture-dependent:

• For attention layers, it chose FLASH_ATTN — a newer backend with Blackwell support ✅
• For MoE routing and expert computation, it chose TRITON — which needs to compile custom kernels ❌

The Triton backend attempted to compile MoE kernels for sm_121a (Blackwell’s compute capability), but PTXAS — NVIDIA’s PTX (Parallel Thread Execution) assembler — hasn’t been updated to recognize this architecture string yet. Even though PyTorch itself supports Blackwell (it detected the GPU correctly), the CUDA compilation toolchain one layer down was still living in the Hopper (SM_90) era.

The Compatibility Stack

The Pattern Emerges: The Matrix of Pain

This BF16 MoE failure was the third vLLM blocker I’ve discovered on GB10. What’s striking is that these are independent failures. You can’t work around one by changing another variable.

Feature Matrix

It’s not a bug — it’s a missing feature matrix.

The Bigger Picture: When Hardware Outpaces Software

There’s a fascinating race happening in the GPU ecosystem. NVIDIA ships hardware with new compute capabilities every 12–18 months. Framework developers scramble to add support. But the stack is deep, and every layer needs updating:

• PyTorch added SM_121 support in version 2.7 (February 2026)
• vLLM added V1 engine improvements in 0.15.1 (January 2026)
• Triton compiler… still catching up
• CUTLASS FP8 templates… still catching up
• PTXAS architecture recognition… still catching up

This isn’t anyone’s fault. It’s the reality of managing a software ecosystem with dozens of moving parts. Each component has its own release cycle, its own maintainers, its own priorities.

The quote I keep returning to: “In theory, there is no difference between theory and practice. In practice, there is.” When NVIDIA’s documentation says “Blackwell supports all previous CUDA operations,” they’re technically correct. The hardware supports it. Whether the software stack supports it is a different question entirely.

The Silver Lining: TensorRT-LLM Just Works

Here’s the plot twist: while vLLM struggles with Blackwell, TensorRT-LLM runs flawlessly. Same hardware, same models, different framework. Why?

TensorRT-LLM is NVIDIA’s first-party inference engine. It ships with:

• Native SM_121 kernel support (because it’s developed in-house)
• NVFP4 quantization (their own format, not CUTLASS’s FP8)
• Optimized MoE routing that doesn’t rely on Triton
• Tight integration with the CUDA toolkit

On our dual-node GB10 cluster, TensorRT-LLM delivers:

• 15.4 tokens/sec on Qwen3–235B (dual-node TP=2, NVFP4)
• 31.4 tokens/sec on Qwen3-Next-80B (single-node, NVFP4)

These aren’t academic benchmarks. These are production workloads, running stable, day after day. While vLLM fought me for 15 hours, TensorRT-LLM handled the same workload in minutes.

Lessons from the Trenches

Observation #1: Backend selection matters more than precision.

I started this test thinking precision was the limiting factor (FP8 vs BF16). I was wrong. The backend chosen for each operation type is what determines compatibility. FlashAttention worked. Triton didn’t. The precision was irrelevant.

Observation #2: “Latest version” doesn’t mean “fully supported.”

We’re running PyTorch 2.9.1, vLLM 0.15.1, CUDA 13.0 — bleeding edge across the board. Yet fundamental operations fail because one component (PTXAS) hasn’t caught up. Version numbers are necessary but not sufficient for compatibility.

Observation #3: Error messages tell you where things broke, not why.

The error said “ptxas fatal: Value ‘sm_121a’ is not defined.” A junior engineer might think: “I’ll update ptxas.” But ptxas ships with CUDA, and CUDA doesn’t update ptxas for pre-release architectures. The real fix requires Triton to generate PTX that ptxas can handle — or bypass ptxas entirely with a different backend. The error is the symptom, not the diagnosis.

Looking Forward: The Inference Cambrian Explosion

Despite today’s failure, I’m optimistic. Here’s why:

The ML inference ecosystem is exploding with innovation. Five years ago, we had one option: run PyTorch in eval mode. Today we have vLLM, TensorRT-LLM, llama.cpp, SGLang, and a dozen others.

This competition drives progress. Six months ago, none of these frameworks supported Blackwell. Today, one does (TensorRT-LLM). Six months from now, several will. The vLLM team is aware of SM_121 limitations — they’re just prioritizing the 99% of users on Hopper and Ampere/Ada first. That’s rational engineering.

And when vLLM does add Blackwell support, I’ll test it again. I’ll download another 57GB model. I’ll watch the progress bars tick up. I’ll monitor GPU memory. I’ll read the logs line by line. Because that’s how you push the boundaries of what’s possible.

Epilogue: What We Learned

Today’s test answered the question: “Can vLLM run MoE models in BF16 on Blackwell?”

Answer: Not yet. But the journey taught us:

• Where the software stack has gaps (Triton/PTXAS compatibility)
• Which backends work (FlashAttention ✅, Triton MoE ❌)
• Why architectural boundaries matter (SM_121 is fundamentally new)
• What production-ready means (TensorRT-LLM’s first-party advantage)

Next week, I’ll be fine-tuning Qwen3–7B on financial time-series data. When deployment time comes, I won’t hesitate: TensorRT-LLM, single-node, NVFP4 quantization. I’ve learned to use the tool that works, not the tool I wish worked.

But I’ll keep one eye on vLLM’s release notes. Because the day they ship SM_121 support, I’ll be first in line to test it.

The bleeding edge cuts both ways. But that’s where the interesting problems live.

Technical Specifications:

• Hardware: DGX Spark with GB10 (Blackwell), SM_121 compute capability, 120GB unified memory
• Software: PyTorch 2.9.1+cu128, vLLM 0.15.1, CUDA 13.0, NVIDIA Driver 580.126.09
• Model: Qwen/Qwen3–30B-A3B-Instruct-2507 (57GB BF16, 16 safetensors shards)
• Test Duration: ~15 minutes (9 min download, 4 min loading, 2 min compilation attempt)
• Result: PTXAS compilation failure in Triton MoE backend