LoRA fine-tuning of Qwen3.5 on NVIDIA DGX Spark

This article covers the complete environment setup for bf16 LoRA fine-tuning of Qwen3.5–35B-A3B on a single NVIDIA DGX Spark (GB10 Superchip), including Docker configuration, dependency pinning, unified memory OOM workarounds, distributed environment cleanup, dataset quirks, training configuration, and inference deployment.

Qwen3.5 MoE fine-tuning on unified memory architecture

Hardware Context

The DGX Spark is built around a Blackwell GB10 Grace Blackwell Superchip:

• 128 GB unified LPDDR5x memory shared between CPU and GPU at 273 GB/s bandwidth via NVLink-C2C interconnect
• Linux kernel sees approximately 119 GB of usable RAM; ~9 GB reserved by GPU firmware
• PyTorch reports 119.635 GB max CUDA memory (~124 GB free at idle)
• cudaMemGetInfo underreports available memory on UMA systems (per NVIDIA's DGX Spark Porting Guide) — do not rely on it for capacity planning
• CPU: 20-core ARM Grace processor (aarch64)
• Single-GPU system — no multi-GPU distributed training
• CUDA compute capability: 12.1 (sm_121)

Model being trained — Qwen3.5–35B-A3B:

• Model class: Qwen3_5MoeForConditionalGeneration
• 35B total parameters (~3B active per forward pass); 256 experts per MoE layer
• Vision-language architecture; text-only training requires tokenizer unwrapping (see below)
• ~67 GB on disk in bf16, split across 14 shard files; ~74 GB in-memory (includes CUDA tensor overhead, padding, alignment)

Confirmed Working Software Stack

Component        | Version / Notes
-----------------|---------------------------------------------------------------
Python           | 3.12
PyTorch          | 2.11.0+cu130 (CUDA 13.0, aarch64)
Transformers     | 5.2.0 (with Unsloth 2026.3.4) or 5.3.0 (with Unsloth 2026.3.17)
PEFT             | 0.18.1
Unsloth          | 2026.3.4 or 2026.3.17 (git main)
bitsandbytes     | Required for AdamW 8-bit optimizer (not used for quantization)
xformers         | Compiled from source for sm_121
wandb            | For training logging

Transformers compatibility note: Transformers version must match the Unsloth release in use. transformers v4.x does not recognize the qwen3_5_moe architecture at all. Always verify against the Unsloth compatibility matrix.

Docker Base Image

FROM nvcr.io/nvidia/pytorch:25.10-py3

This image targets the Blackwell GB10 (sm_121 / compute_121). It is the required base because:

• xformers must be compiled from source for sm_121 (no pre-built wheels cover this architecture)
• The base image provides the correct CUDA toolkit version for that compilation

Dependency Management

Dependencies are managed with uv, with a uv.lock file for fully locked, reproducible versions.

# pyproject.toml — pin transformers to match your Unsloth release:
# Unsloth 2026.3.4  → transformers = "==5.2.0"
# Unsloth 2026.3.17 → transformers = "==5.3.0"
transformers = "==5.2.0"

Installation order note: When installing Unsloth from git, unsloth_zoo must be installed before unsloth — it is not pulled in automatically.

The UMA OOM Problem and Solution

Why Standard Loading Fails

The standard HuggingFace safetensors loader uses memory-mapped I/O (mmap). On a conventional discrete-GPU system, mmap pages live in CPU RAM while CUDA tensors live in VRAM — two separate pools. On DGX Spark’s unified memory pool, both the mmap page cache and the materialized CUDA tensors compete for the same ~119 GB of physical memory.

The OOM kill occurs at exactly 66% of weight loading (~680 of 1026 parameter tensors), corresponding to ~119/134 GB consumed: mmap-backed pages for already-loaded shards remain in the page cache while CUDA tensors from those same shards have been materialized.

Standard workarounds that do not resolve this (none change the underlying mmap behavior):

• device_map='sequential'
• offload_state_dict=True
• SAFETENSORS_FAST_GPU=1

QLoRA (4-bit) makes it worse, not better: load_in_4bit=True OOMs even earlier (~4%) because bitsandbytes quantization creates large intermediate buffers on top of mmap overhead. Additionally, bitsandbytes 4-bit loading triggers device_map='auto' with CPU offload, which is incompatible with accelerate's distributed-mode checks on this platform. Unsloth also explicitly warns against QLoRA for Qwen3.5 MoE due to quality degradation. Use bf16 LoRA.

The _EagerSafeOpen Fix

The fix is a custom safetensors loader that:

1. Loads each shard eagerly and directly to CUDA (bypassing persistent mmap)
2. Calls posix_fadvise(POSIX_FADV_DONTNEED) to evict that shard's pages from the page cache immediately after tensors are materialized on CUDA
3. Is injected via monkey-patching transformers.modeling_utils.safe_open to intercept all HuggingFace loading calls transparently

Memory profile with this fix:

Phase                               | Memory
------------------------------------|-------------------------
Peak during model loading           | ~72 GB (5 GB transient mmap/shard + 67 GB accumulated CUDA tensors)
After model load (free)             | ~52 GB
LoRA adapters                       | ~0.5 GB
AdamW 8-bit optimizer states        | ~1 GB
Gradients + activations             | ~3–5 GB
Remaining headroom                  | ~46+ GB
Total training footprint (Unsloth)  | ~74 GB

import ctypes
import safetensors
import transformers.modeling_utils

POSIX_FADV_DONTNEED = 4

def _eager_cuda_safe_open(path, framework, device):
    """
    Custom safe_open replacement:
    1. Loads each shard eagerly and directly to CUDA.
    2. Calls posix_fadvise(POSIX_FADV_DONTNEED) to evict the shard's
       pages from the Linux page cache immediately after materialization.
    """
    libc = ctypes.CDLL("libc.so.6", use_errno=True)

    with open(path, "rb") as f:
        fd = f.fileno()
        with safetensors.safe_open(path, framework=framework, device=device) as st:
            for key in st.keys():
                tensor = st.get_tensor(key)
                yield key, tensor
        # Evict this shard's pages from the page cache
        file_size = f.seek(0, 2)
        libc.posix_fadvise(fd, 0, file_size, POSIX_FADV_DONTNEED)

# Monkey-patch transformers to intercept all HuggingFace loading calls
transformers.modeling_utils.safe_open = _eager_cuda_safe_open

Runtime Environment Variables

Set the following before launching any training script:

# Reduce CUDA allocator fragmentation (important near UMA memory ceiling)
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
# Prevent Rust tokenizer threading issues in forked processes
export TOKENIZERS_PARALLELISM=false

Distributed Training Environment Cleanup

The DGX Spark is a single-GPU system. Accelerate, DeepSpeed, FSDP, and NCCL/MPI all rely on environment variables that — if present from prior configurations — cause frameworks to attempt multi-process/multi-node initialization incompatible with a single-GPU setup.

Variables to Unset

• WORLD_SIZE, RANK, LOCAL_RANK, MASTER_ADDR, MASTER_PORT
• All NCCL_* variables
• All OMPI_* and MPI_* variables

Additional Explicit Disabling Required

Even without WORLD_SIZE, accelerate may attempt FSDP or DeepSpeed if ~/.cache/huggingface/accelerate/default_config.yaml exists. Setting ACCELERATE_USE_FSDP=0 and ACCELERATE_USE_DEEPSPEED=0 overrides any cached config and is required in addition to unsetting process-rank variables.

Shell Setup (run before any training launch)

# Disable distributed training backends
export ACCELERATE_USE_FSDP=0
export ACCELERATE_USE_DEEPSPEED=0
# Remove stale distributed coordination variables
unset WORLD_SIZE
unset RANK
unset LOCAL_RANK
unset MASTER_ADDR
unset MASTER_PORT
# Also unset any NCCL_* and MPI_* / OMPI_* variables present in your shell

# Equivalent in Python (for training scripts)
import os
os.environ["ACCELERATE_USE_FSDP"] = "0"
os.environ["ACCELERATE_USE_DEEPSPEED"] = "0"
for var in ["WORLD_SIZE", "RANK", "LOCAL_RANK", "MASTER_ADDR", "MASTER_PORT"]:
    os.environ.pop(var, None)
# Also strip NCCL_* and MPI_* / OMPI_* as appropriate

Auditing Your Shell

env | grep -E "^(WORLD_SIZE|RANK|LOCAL_RANK|MASTER_ADDR|MASTER_PORT)="
env | grep "^NCCL_"
env | grep -E "^(OMPI_|MPI_)"
env | grep "^ACCELERATE_USE_"

Do Not Use device_map='auto' Manually

Within training scripts, avoid setting device_map='auto'. Use explicit device placement or allow Unsloth's FastModel.from_pretrained() to handle device assignment internally. device_map='auto' invokes accelerate's get_balanced_memory, which performs distributed-mode checks incompatible with this platform.

Model Loading: API and Configuration

Use FastModel, Not FastLanguageModel

FastLanguageModel is for dense models only. For MoE architectures like Qwen3.5-35B-A3B, it produces warnings about unsupported layer types and handles expert layers incorrectly.

from unsloth import FastModel  # correct for MoE
# NOT: from unsloth import FastLanguageModel
model, tokenizer = FastModel.from_pretrained(
    model_name=model_id,
    dtype=torch.bfloat16,
    load_in_4bit=False,  # bf16, not QLoRA
)

Base vs Instruction-Tuned Model

Fine-tuning on the base (pre-trained) model is preferred. RLHF training on the instruction-tuned variant can conflict with domain-specific fine-tuning objectives.

Model Cache Persistence

The 67 GB model download should only happen once. Mount the model cache directory as a persistent volume across container restarts to avoid re-downloading.

Note: HuggingFace Hub download speed on DGX Spark is approximately 30 KB/s (vs ~1 MB/s for general network traffic). Plan accordingly or pre-cache weights.

Runtime Monkey-Patches

fix_untrained_tokens → No-op

import unsloth.tokenizer_utils
# Qwen3.5 vocabulary is fully trained; no unused token slots - skip this function
unsloth.tokenizer_utils.fix_untrained_tokens = lambda *args, **kwargs: None

Text-Only Tokenizer Access for Qwen3VLProcessor

For text-only fine-tuning of Qwen3.5–35B-A3B (a vision-language model), access the underlying text tokenizer directly to avoid triggering image-processing code paths. Also required because generate() expects a torch.Tensor, not a BatchEncoding.

from transformers import Qwen3VLProcessor
processor = Qwen3VLProcessor.from_pretrained(model_id)
tokenizer = processor.tokenizer  # unwrap for text-only tasks
# Generic unwrap pattern:
tokenizer = getattr(tokenizer, 'tokenizer', tokenizer)

Unsloth Telemetry Timeout (Restricted Networks)

Unsloth’s stats check times out after 120 seconds on restricted networks. The fix requires editing vision.py in the Unsloth source directly — Python's from X import Y creates a local binding that survives module-level monkey-patching at runtime.

LoRA Configuration

Key Constraints for MoE

• lora_dropout must be 0.0: PEFT's ParamWrapper (used for MoE expert layers) does not support nonzero dropout. Any nonzero value causes errors or silently broken training. As a secondary benefit, lora_dropout=0.0 enables Unsloth's fast patching across all layers.
• Use use_gradient_checkpointing="unsloth" for Unsloth's memory-efficient implementation; do not use standard gradient checkpointing.

Validated Configuration

model = FastModel.get_peft_model(
    model,
    r=16,                           # r=32 also validated; r=64 recommended if F1 needs improvement
    lora_alpha=16,                  # r=32 config uses lora_alpha=32 (scaling = 1.0)
    lora_dropout=0.0,               # must be 0.0 for MoE with PEFT ParamWrapper
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    bias="none",
    use_gradient_checkpointing="unsloth",
)

Trainable Parameter Counts at r=32

• Trainable parameters: 1,862,270,976 out of 36,974,764,032 total (5.04%)
• Adapter weights file size: ~7.4 GB (adapter_model.safetensors)
• To improve F1, increase rank: r=64 roughly doubles trainable parameters

Training Configuration

Use SFTConfig + SFTTrainer, Not TrainingArguments

SFTTrainer must be paired with SFTConfig. SFT-specific fields like dataset_text_field may be silently ignored when using the plain TrainingArguments class.

from trl import SFTConfig, SFTTrainer
training_args = SFTConfig(
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,   # effective batch size = 8
    num_train_epochs=3,
    learning_rate=2e-4,
    lr_scheduler_type="linear",
    warmup_steps=5,                  # warmup_ratio is deprecated in transformers 5.x
    weight_decay=0.01,
    bf16=True,
    fp16=False,
    optim="adamw_8bit",
    max_seq_length=2048,             # safe upper limit; 8192 causes OOM on backward pass
    packing=False,                   # disable for entity extraction / structured tasks
    dataset_num_proc=1,              # avoid fork deadlocks (see below)
)

warmup_ratio Deprecation

In Transformers 5.x, warmup_ratio is deprecated. Use warmup_steps instead.

max_seq_length Hard Limit

max_seq_length=2048 is the confirmed safe limit for a 37B MoE model on the GB10. Values such as 8192 cause OOM during the backward pass.

Sequence Packing

Disable sequence packing for entity extraction and similar structured tasks. Packing causes the model to see multiple documents as a single sequence, producing cross-document entity confusion.

Dataset and Tokenizer Fixes

# Avoid fork deadlocks in dataset.map()
dataset = dataset.map(formatting_fn, num_proc=1)

Training Data Format

Training data is stored as JSONL in ShareGPT/ChatML conversation format:

{"conversations": [{"from": "human", "value": "What is 2 + 2?"}, {"from": "gpt", "value": "4"}]}
{"conversations": [{"from": "human", "value": "Explain LoRA fine-tuning."}, {"from": "gpt", "value": "LoRA adds low-rank adapter matrices to frozen model weights..."}]}

Observed Training Results (301-sample NER Dataset, 3 Epochs)

• Dataset: 301 samples in OpenAI chat format; avg ~2,046 tokens/sample, P95 ~3,594, max ~7,731
• Train/eval split: 271/30 samples (90/10%), seed 42
• Total training steps: 102
• Total training time: 2 hours 24 minutes on a single GB10
• Loss: start ~0.648 → step 50 eval loss 0.568 → final train loss ~0.521, final eval loss 0.568

Inference: llama.cpp (Recommended)

Unsloth’s fast_inference is disabled on the DGX Spark platform. Without it, generation falls back to HuggingFace Transformers at approximately ~2 tok/s — effectively unusable for production. Use llama.cpp instead.

Inference Performance Comparison

Backend                        | Throughput     | Avg Inference Time | JSON Valid Rate     | F1
------------------------------|----------------|--------------------|---------------------|------
llama.cpp (q4_k_m)            | ~42.3 tok/s    | 21.4s              | 100% (30/30)        | 0.871
Unsloth / Transformers        | ~2 tok/s       | 216.1s             | 93.3% (28/30)       | 0.862

llama.cpp is approximately 10× faster in latency and 21× higher in throughput on this hardware. Throughput range across 30 eval samples: 40.5–42.7 tok/s.

GGUF Export

Export the merged model to GGUF with q4_k_m quantization:

• Quantized model size: approximately 20 GB (q4_k_m, 4-bit k-quant medium)

llama.cpp Build Flags for GB10 (sm_121)

cmake .. \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_F16=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES=121 \
  -DGGML_NATIVE=ON
# WARNING: Do NOT set -DGGML_CUDA_FORCE_CUBLAS=ON on GB10 Blackwell — causes garbage output

llama.cpp Server Invocation

llama-server \
  --no-mmap \
  --chat-template chatml \
  --ctx-size 16384 \
  -ngl 999 \
  -fa 1

Key flag notes:

• --no-mmap: Cuts model load time from ~104s to ~22s
• --chat-template chatml (not --jinja): The --jinja flag picks up the embedded Qwen3.5 thinking template and wastes tokens on reasoning_content
• --ctx-size 16384 vs 32768: Increasing context to 32768 drops decode speed by approximately 30%