Running Mistral Small 4 (119B, NVFP4) Locally on a DGX Spark

A practical guide to serving a 119B parameter model on NVIDIA’s compact Blackwell workstation — including the quirks of the consumer GB10 GPU

Not a medium member? you can read the full article for free here with this link

When Mistral AI released Mistral Small 4 in March 2026, I immediately wanted to run it locally on my DGX Spark. On paper, the math works: the model weighs ~66 GiB in NVFP4 quantization, and the DGX Spark has 128 GiB of unified memory. In practice, getting there required navigating a few surprises specific to the Blackwell consumer GPU. Here’s everything I learned.

The Model: Mistral Small 4

mistralai/Mistral-Small-4-119B-2603-NVFP4 is one of the most capable open-weight models released so far. What makes it unusual is that it unifies three model families in a single checkpoint: Instruct, Reasoning (formerly Magistral), and Devstral. You don't need to choose — it does everything.

Under the hood, it’s a Mixture-of-Experts architecture:

• 119B total parameters, but only 6.5B active per token thanks to MoE (128 experts, 4 active)
• 256k context window
• Multimodal: text and image inputs, text output
• Configurable reasoning: toggle reasoning effort per request — from instant reply to deep chain-of-thought
• Multilingual: English, French, Spanish, German, Chinese, Japanese, Korean, Arabic, and more
• Apache 2.0 license: free for commercial use

This particular checkpoint is a post-training activation quantized version using NVFP4, produced in collaboration between Mistral AI, vLLM, and Red Hat via llm-compressor. Compared to Mistral Small 3, it delivers 40% lower end-to-end completion time and 3x higher throughput in optimized configurations.

The Hardware: DGX Spark and Its Blackwell Quirk

The DGX Spark is NVIDIA’s compact AI workstation, built around the GB10 Grace Blackwell Superchip. It’s a remarkable machine in a small form factor, with 128 GiB of unified CPU+GPU memory — enough to fit large quantized models entirely in memory.

The important caveat: the GB10 uses SM121, not SM100. SM100 is the compute capability of datacenter Blackwell GPUs (B200, GB200). SM121 is the consumer/workstation variant. This distinction matters a lot for CUDA kernel compatibility, as we’ll see below.

Step 1: Download the Model

hf download mistralai/Mistral-Small-4-119B-2603-NVFP4

The model (~66 GiB) will be stored in ~/.cache/huggingface/hub/. On my setup, the weights live on an external USB SSD (~475 MB/s read), which means about 10 minutes of loading on every startup. A fast NVMe would be better, but the DGX Spark's internal NVMe is mostly occupied by the system.

Step 2: Install vLLM

The official vLLM repo doesn’t yet fully support Mistral v15 tokenizer parsing. Mistral AI provides a patched fork specifically for this model:

git clone --branch fix_mistral_parsing https://github.com/juliendenize/vllm.git vllm-mistral
cd vllm-mistral

This branch is expected to be merged into vLLM main soon. Track progress here.

Create the virtual environment

uv venv
source .venv/bin/activate

Install vLLM with precompiled kernels

VLLM_USE_PRECOMPILED=1 uv pip install --editable .

Install PyTorch for CUDA 13

The DGX Spark runs CUDA 13.0, which requires a specific PyTorch build:

uv pip install --index-url https://download.pytorch.org/whl/cu130 torch==2.10.0+cu130
uv pip install --index-url https://download.pytorch.org/whl/cu130 torchvision

Install Transformers and mistral_common

The Mistral v15 tokenizer requires a recent version of mistral_common:

uv pip install git+https://github.com/huggingface/transformers.git
pip install --upgrade mistral_common

Install FlashInfer

uv pip install flashinfer-python flashinfer-cubin
pip install flashinfer-jit-cache --index-url https://flashinfer.ai/whl/cu130

Step 3: Configure the systemd Service

I run vLLM as a persistent system service so it starts automatically on boot and restarts on failure.

sudo systemctl edit --force --full vllm.service

[Unit]
Description=vLLM Inference Server
After=network.target

[Service]
User=sb
Group=sb
WorkingDirectory=/mnt/data/sb/projects/vllm-mistral
Environment="PATH=/mnt/data/sb/projects/vllm-mistral/.venv/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
Environment="LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64"
Environment="TORCH_CUDA_ARCH_LIST=12.1a"
Environment="TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas"
Environment="FLASHINFER_JIT_LOG_LEVEL=ERROR"
Environment="TRANSFORMERS_VERBOSITY=error"
Environment="VLLM_SKIP_P2P_CHECK=1"
ExecStart=/mnt/data/sb/projects/vllm-mistral/.venv/bin/vllm serve mistralai/Mistral-Small-4-119B-2603-NVFP4 \
    --max-model-len 262144 \
    --tensor-parallel-size 1 \
    --attention-backend TRITON_MLA \
    --tool-call-parser mistral \
    --enable-auto-tool-choice \
    --reasoning-parser mistral \
    --max-num-batched-tokens 16384 \
    --max-num-seqs 128 \
    --gpu-memory-utilization 0.8 \
    --no-enable-flashinfer-autotune \
    --cudagraph-capture-sizes 1 2 4 8 16 32 64 128 256 \
    --max-cudagraph-capture-size 256
Restart=always
RestartSec=15
StandardOutput=journal
StandardError=journal
SyslogIdentifier=vllm
MemoryMax=120G
MemorySwapMax=0

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now vllm

The Configuration Choices — and Why They Matter

This is where the GB10’s specifics really come into play.

--attention-backend TRITON_MLA

This is the most critical flag. FlashAttention and FlashInfer MLA are compiled for SM100 (datacenter Blackwell). Running them on SM121 causes an immediate cudaErrorIllegalInstruction crash.

Triton solves this by JIT-compiling the attention kernels at runtime for SM121 specifically. It’s slightly slower than FlashAttention on supported hardware, but it’s the only option that works here.

--no-enable-flashinfer-autotune

This flag had the single biggest impact on startup time, and it took me a while to figure out why.

On the second startup, without this flag, vLLM spent over 21 minutes stuck in the init engine phase. What was happening: the FlashInfer autotuner was systematically testing dozens of MoE kernel tactics compiled for SM120 — all of which fail on SM121. It tried every single one before giving up.

With --no-enable-flashinfer-autotune, the logs simply say:

Skipping FlashInfer autotune because it is disabled.

And those 1300 seconds disappear entirely.

--cudagraph-capture-sizes 1 2 4 8 16 32 64 128 256

By default, vLLM captures CUDA graphs for dozens of batch sizes (up to 512, in steps of 8). For a personal or small-team deployment, you’ll rarely need more than a handful of concurrent requests. Limiting capture to 9 power-of-2 sizes reduces graph capture time significantly and saves VRAM.

TORCH_CUDA_ARCH_LIST=12.1a

Tells PyTorch and Triton to compile kernels for SM121. The a suffix is specific to the GB10 consumer variant, distinct from the 12.0 datacenter architecture.

TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas

Points Triton to the CUDA 13 PTX assembler. Without this, Triton may pick up an incompatible version from the system path.

VLLM_SKIP_P2P_CHECK=1

Skips a peer-to-peer GPU connectivity check that can add ~60 seconds on single-GPU setups where it’s irrelevant.

Startup Performance

Here’s what a warm startup looks like (torch.compile cache already populated from the first run):

10:13:14 → service started
10:23:53 → Loading weights took 606.35 seconds
10:25:13 → torch.compile took 3.96 s (cache hit)
10:25:38 → Graph capturing finished in 13 secs
10:25:39 → init engine took 103.80 seconds
10:25:41 → Application startup complete

The weight loading (~10 min) is the dominant cost and is essentially fixed by the storage speed. The torch.compile cache lives at ~/.cache/vllm/torch_compile_cache/ and kicks in automatically from the second startup onward.

Benchmark Results

I ran three benchmark scenarios using vllm bench serve to measure real-world performance.

Solo use (1 concurrent request)

Output token throughput:   27.80 tok/s
Peak output throughput:    29.00 tok/s
Time to First Token (TTFT): 115ms
Time per Output Token:      35ms (~28 tok/s)

115ms TTFT means the response starts appearing almost instantly. At 28 tok/s, the text streams roughly 3x faster than average reading speed — the generation is effectively invisible to the user.

Medium load (5 concurrent requests, 256 output tokens)

Output token throughput:   65 tok/s
Peak output throughput:    75 tok/s
Time to First Token (TTFT): ~1.6s
Time per Output Token:      70ms (~14 tok/s per user)

High load (20 concurrent requests, 1024 output tokens)

Output token throughput:   131 tok/s
Peak output throughput:    166 tok/s
Time to First Token (TTFT): ~1.4s
Time per Output Token:      151ms (~6.5 tok/s per user)

Summary

A few things stand out. First, the MoE architecture pays off under load: total throughput more than doubles going from 5 to 20 concurrent requests, as the model efficiently batches across active experts. Second, even at 20 simultaneous users, TTFT stays under 1.5 seconds. Third, solo TTFT of 115ms is genuinely impressive for a 119B parameter model.

For personal or small team use, this setup delivers a fluid, responsive experience.

Quick Test

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistralai/Mistral-Small-4-119B-2603-NVFP4",
    "messages": [{"role": "user", "content": "What is a Mixture of Experts model?"}],
    "max_tokens": 256
  }'

Optional: Open WebUI

For a browser-based chat interface accessible from another machine on the network:

docker run -d \
  --network=host \
  --name open-webui \
  -e OPENAI_API_BASE_URL=http://localhost:8000/v1 \
  -e OPENAI_API_KEY=none \
  -v open-webui:/app/backend/data \
  --restart always \
  ghcr.io/open-webui/open-webui:main

Available at http://<DGX_IP>:8080.

Known Gotchas

PyTorch SM121 warning — you’ll see this on every startup:

Found GPU0 NVIDIA GB10 which is of cuda capability 12.1.
Minimum and Maximum cuda capability supported by this version of PyTorch is (8.0) - (12.0)

It’s harmless. PyTorch 2.10+cu130 officially supports up to SM 12.0, but works fine on SM 12.1 via the TORCH_CUDA_ARCH_LIST=12.1a flag.

safetensors repo error — also harmless:

ERROR: 'mistralai/Mistral-Small-4-119B-2603-NVFP4' is not a safetensors repo.

This model uses Mistral’s own consolidated.safetensors format rather than the standard HuggingFace layout. vLLM handles the fallback correctly.

Tensorizer — if you’re hoping to use tensorizer to speed up weight loading, it’s currently incompatible with NVFP4 / compressed-tensors quantization. I tried; it doesn’t work.

Conclusion

Running a 119B parameter model locally is entirely feasible on a DGX Spark — but the consumer Blackwell GB10 (SM121) has its own set of compatibility quirks that aren’t well documented yet. The three things that made it work:

1. Use TRITON_MLA as the attention backend — the only option compatible with SM121
2. Disable the FlashInfer autotuner — this alone saves over 20 minutes on repeated startups
3. Limit CUDA graph capture sizes — cuts graph capture from ~44s down to ~13s for personal use

Once running, the server is stable and responsive. The ~12 minute cold start is dominated by loading 66 GiB from a USB SSD — something a faster storage solution would improve significantly.

If you’re working on a DGX Spark and run into issues, feel free to reach out. Happy to compare notes.