Quantization for LLMs, VLMs, and Diffusion Models: A Practical Production Guide

Which method to use, when to use it, and how to ship it without wrecking quality.

A 70B parameter model in FP16 needs about 140 GB of VRAM for weights alone. That is a multi-GPU deployment before you have served a single token. Quantize the same model to 4 bits and the weights fit in roughly 35 GB, which means one workstation GPU, a fraction of the serving cost, and in many memory-bound scenarios, faster generation too.

That single lever, bits per parameter, decides more about your deployment architecture than almost any other choice you make after training. It decides whether you need a cluster or a desktop, whether your unit economics work, and whether your fine-tuned model ever leaves the notebook it was trained in.

The problem is that the quantization landscape looks like alphabet soup. GPTQ, AWQ, GGUF, NF4, FP8, SmoothQuant, SVDQuant. Half the tutorials online recommend libraries that were deprecated last year. And almost nothing covers the models I care about most these days: vision-language models and diffusion models, where the rules genuinely change.

This guide is the article I wanted when I started quantizing models for production. It answers one question throughout: given your hardware and your deployment goal, which quantization method should you choose? Everything else exists to support that answer.

The cheat sheet, up front

If you read nothing else, read this table. The rest of the article explains why these recommendations hold and where they break.

Quantization in one page

Quantization maps high-precision numbers (FP32, FP16, BF16) to a smaller set of low-precision values (INT8, FP8, INT4). Fewer bits per number means less memory, less data movement, and on supported hardware, faster math. The cost is rounding error, and the entire craft of quantization is deciding where that error is affordable.

Beginners usually think of quantization as one switch you flip on a model. It is closer to three switches, because a transformer at inference time has three distinct tensor populations, each with its own memory bill and its own tolerance for error.

Weights are fixed after training, which makes them the easy target. You can analyze them offline, find the sensitive channels, and spend as much compute as you like getting the rounding right. Nearly every method you have heard of (GPTQ, AWQ, GGUF, NF4) is weight-only quantization: weights are stored in 4 bits, then dequantized to FP16 on the fly for each matrix multiply.

Activations are produced fresh for every input, so they must be quantized at runtime with no second chances. They are also plagued by outliers: rare channels with values hundreds of times larger than their neighbors, which single-handedly destroy naive quantization. Methods like SmoothQuant exist specifically to migrate that outlier difficulty from activations into weights, where it can be handled offline.

The KV cache stores attention keys and values for every token in context. At short context it is a rounding error; at 128K context with a big batch it can exceed the weights themselves. Modern inference engines can hold it in FP8, which doubles the context or batch you fit in the same VRAM. In vLLM that is one flag, and I will show it in the implementation section.

There is a second distinction hiding here that explains most of the confusion about quantization speedups. Weight-only INT4 makes the model smaller, but the arithmetic still happens in FP16 after dequantization. So it accelerates workloads that are memory-bound: decoding, small batches, local inference, where the GPU spends its time streaming weights from VRAM rather than computing. Weight-plus-activation formats like FP8 let the tensor cores do the math directly in low precision, which accelerates compute-bound workloads: prefill and high-batch serving. Casper Hansen, the author of AutoAWQ, documented this plainly: at higher batch sizes a W4A16 model gains no speedup because dequantization overhead eats the win.

Keep this pair of regimes in mind. It is the physics underneath every recommendation in the next section.

The decision framework

Here is the whole article in one flowchart. Find your deployment scenario, follow the branch, and you have a defensible starting point. The sections after this explain the reasoning so you can deviate intelligently.

A few notes the boxes cannot hold:

Data center serving splits on hardware generation. FP8 needs native tensor core support, which arrived with Hopper (H100) and Ada Lovelace. On those chips, FP8 W8A8 is the closest thing to a free lunch in this field: roughly half the memory of BF16, higher throughput under load, and quality recovery routinely above 99%. On Ampere (A100), FP8 execution is emulated at best, so INT4 weight-only via AWQ or GPTQ with vLLM’s Marlin kernels is the practical choice.

Local inference belongs to GGUF. llama.cpp and its GGUF format were built for CPUs and Apple Silicon, with a mature ecosystem (Ollama, LM Studio) on top. The k-quant series gives you a quality dial; Q4_K_M is the community default for a reason, and Q8_0 is nearly indistinguishable from FP16 if you have the memory.

Fine-tuning on a budget means QLoRA. Freeze the base model in 4-bit NF4, train small LoRA adapters in BF16 on top. The QLoRA paper demonstrated fine-tuning a 65B model on a single 48 GB GPU while matching 16-bit fine-tuning quality on their benchmarks.

When quality is paramount, do nothing. BF16 remains the reference. Quantization is a tradeoff you take when memory, cost, or latency force your hand, which in production is almost always.

The three axes of quantization

Every named method is a bundle of three independent decisions. Once you see the axes, the zoo of acronyms collapses into a small grid.

Axis 1: when. Post-training quantization (PTQ) takes a finished model and converts it, usually in minutes to hours with a small calibration set. Quantization-aware training (QAT) simulates low precision during training so the model learns around the rounding error. QAT gives the best quality at a given bit width but costs training compute, which is why it was historically rare. That is changing: Google ships official QAT checkpoints of Gemma 3 that hold up at 4 bits far better than PTQ conversions of the same model. Unless you are the one training the model, though, you live in PTQ land.

Axis 2: what. Weight-only (W4A16, W8A16) versus weight-and-activation (W8A8, whether INT8 or FP8). This is the memory-bound versus compute-bound story from earlier, crystallized into notation.

Axis 3: how scales are set. Every quantized tensor needs a scale factor mapping its low-precision integers back to real values. Static schemes fix scales ahead of time using calibration data. Dynamic schemes compute them at runtime per token or per tensor, which adapts better to unusual inputs at a small speed cost. Weights are always static (they never change); the choice matters for activations and explains why calibration data quality matters so much for static W8A8 methods.

AWQ, in this grid, is PTQ + weight-only + static. vLLM’s FP8 is PTQ + W8A8. Gemma 3 QAT is QAT + weight-only. Nothing you meet in the wild falls outside the grid.

The methods that matter, tiered by adoption

Not all methods deserve equal attention. Here is the field organized by what you will actually encounter in production, followed by the comparison table.

The production workhorses

AWQ (Activation-aware Weight Quantization). The insight: roughly 1% of weight channels matter far more than the rest, and you can identify them by looking at activation magnitudes. AWQ scales those salient channels up before quantization to protect them. It needs only a light calibration pass, generalizes well, and has first-class kernel support in vLLM. For INT4 serving on GPUs, this is my default. One important note on tooling: the AutoAWQ library that popularized the method is officially deprecated. It was maintained by a single developer in his free time, sustaining a project with over 2 million downloads and 7,000+ dependent models on Hugging Face, and in 2025 he archived it. The vLLM project adopted the functionality into llm-compressor, which is where new AWQ work happens. Plenty of older tutorials (and older model cards) still reference AutoAWQ; it still functions against the library versions it was last tested with (Torch 2.6.0, Transformers 4.51.3), but do new work in llm-compressor.

GPTQ. The academic ancestor of practical INT4. It quantizes weights column by column, using second-order (Hessian) information to update remaining weights and compensate for each rounding error. Quality is comparable to AWQ; it is somewhat more sensitive to calibration data because of how deeply that data shapes the error compensation. Same tooling story as AWQ: the original AutoGPTQ library was archived in April 2025, with GPTQModel and llm-compressor as its successors. Thousands of pre-quantized GPTQ checkpoints on the Hub remain perfectly serviceable.

GGUF / llama.cpp k-quants. A file format plus a family of quantization schemes (Q2_K through Q8_0) designed for llama.cpp. The k-quants use block-wise quantization with clever bit allocation, spending more bits on more important blocks. If your target is a CPU, a Mac, or the Ollama ecosystem, this is your format regardless of any other consideration, because it is the format that hardware path supports.

bitsandbytes NF4. The quantization engine behind QLoRA and behind load_in_4bit=True in Transformers. NF4 (NormalFloat4) is a 4-bit data type whose quantization levels are spaced for normally distributed weights, which real weights approximately are. It quantizes on the fly at load time with no calibration step, making it the most convenient option in existence and the standard base for parameter-efficient fine-tuning. For pure inference throughput, dedicated formats like AWQ with proper kernels serve faster.

Data center scale

FP8. Less an algorithm than a data type with hardware behind it. On Hopper and Ada, FP8 tensor cores execute matrix math at low precision natively, so both weights and activations drop to 8 bits with quality recovery that is routinely above 99% on standard evaluations. Being a float format, FP8 handles activation outliers far more gracefully than INT8. If you serve at scale on H100-class hardware, this should be your starting assumption.

SmoothQuant. The INT8 W8A8 answer for hardware without FP8. It mathematically migrates quantization difficulty from activations (where outliers live) into weights (which tolerate it), enabling accurate INT8 compute on Ampere. Available as a modifier in llm-compressor and often composed with GPTQ.

Emerging and specialized

SVDQuant / Nunchaku brings 4-bit weights AND activations to diffusion models by absorbing outliers into a small high-precision low-rank branch. More on this in the diffusion section, where it is the headline. HQQ quantizes without any calibration data at all, trading a little quality for speed and robustness. AQLM and QuIP# push toward 2 to 3 bits with vector quantization and lattice codebooks; impressive research, rarely worth the complexity in production today. BitNet trains 1.58-bit models from scratch and belongs to the future section.

Hardware compatibility, the part everyone skips

Methods exist because hardware differs. A quantization format without a fast kernel on your GPU is a decompression tax, and a 4-bit model with a bad kernel can genuinely run slower than FP16. Check this table before you commit.

The classic trap lives on Ampere. vLLM will accept FP8 checkpoints on an A100 by running weight-only FP8 with FP16 compute (a compatibility path), and people conclude they are getting FP8 acceleration. They are getting FP8 storage. The tensor cores that make FP8 fast simply do not exist on that silicon. When throughput matters on Ampere, benchmark INT4 Marlin against your FP8-on-A100 setup and let the numbers decide.

Quantizing VLMs: respect the architecture

Vision-language models look like one model but quantize like three. Every mainstream VLM (LLaVA-style architectures, Qwen-VL, InternVL) is a pipeline: a vision encoder turns the image into embeddings, a projector maps those into the language model’s space, and an LLM does everything else.

The parameter budget is wildly lopsided. The vision encoder is typically a few hundred million to a couple billion parameters, while the LLM backbone carries the rest, often 90% or more of the total. So the memory savings live almost entirely in the LLM, and conveniently, the LLM is also the component we know how to quantize well.

The vision encoder is a different story, for two reasons. First, quantizing it saves you very little, so the risk-reward is poor from the start. Second, its errors are uniquely positioned to hurt: the visual embeddings feed every subsequent token the model generates, so a corrupted image representation degrades the entire answer rather than one word of it. Vision transformer activations also carry heavy outliers that vary with input resolution, which makes calibration less reliable than for text. Research on VLM-specific quantization consistently finds cross-modal components more fragile than language-only layers.

The playbook that follows from this:

• LLM backbone: quantize aggressively. INT4 AWQ/GPTQ or FP8, exactly as you would a text-only model. This is standard practice in official quantized releases of open VLMs.
• Projector: small enough that quantizing it is optional. INT8 if you want uniformity; validate on image tasks either way.
• Vision encoder: keep it FP16/BF16 by default. It costs you a gigabyte or two, and it buys you stability.

One practical note: llm-compressor supports this pattern directly. Its multimodal examples quantize the language layers while listing the vision tower in the ignore list, so the recommended architecture-aware recipe is also the path of least resistance.

Evaluate VLMs on visual tasks after quantizing, and include at least one task requiring fine detail (OCR, chart reading, document QA). Those degrade first, well before general image captioning shows any change.

Quantizing diffusion models: yes, and the rules change

Short answer to the question that motivated this article: quantization absolutely applies to Stable Diffusion, SDXL, and Flux, it is increasingly standard for local deployment, and it fails in ways LLM intuition will not predict.

A diffusion pipeline has three components with radically different tolerances.

Text encoders: quantize freely. Flux and SD3 ship with CLIP plus a T5-XXL encoder, and T5-XXL alone is about 4.7B parameters, a big slice of the total memory. It runs once per generation, its output conditions the image rather than composing it, and in practice it tolerates INT8 and even INT4 GGUF quantization with minimal impact on prompt adherence. The community figured this out fast: quantized T5 checkpoints are everywhere in the ComfyUI ecosystem.

The DiT or UNet: quantize carefully. This is the bulk of compute and it runs 20 to 50 times per image. Here is the property that separates diffusion from LLMs: the denoiser feeds its own output back to itself. A small quantization error in one LLM forward pass produces one slightly-off logit distribution. The same per-step error in a denoiser is applied dozens of times and compounds, surfacing as texture artifacts, color shifts, and composition drift. Naive INT8 PTQ visibly damages outputs where the same treatment on an LLM would be nearly free. What works: FP8 on Hopper/Ada-class hardware holds up well and is supported in TensorRT and ComfyUI workflows. For 4-bit, SVDQuant is the current state of the art: it absorbs outliers in both weights and activations into a 16-bit low-rank branch, and its Nunchaku engine reports around 3.5x memory reduction on the 12B Flux.1 model with quality maintained, and large speedups over weight-only baselines on consumer GPUs. Timestep-aware calibration (sampling calibration data across the denoising trajectory) is what separates diffusion-specific methods from naive ports of LLM techniques.

The VAE: leave it alone. The VAE decoder is small, runs once, and touches every pixel of the final image. Quantizing it saves almost nothing and risks global artifacts. Keep it FP16. (If you have VRAM pressure from the VAE at high resolutions, tiled decoding solves the real problem.)

Why this matters beyond hobbyists running Flux on a 4090: image generation at product scale is brutally compute-intensive, and a 3x memory reduction on the DiT changes what hardware tier you can serve from. If you run a generative media product, DiT quantization is a line item on your margin.

Implementation guide

One deep walkthrough (AWQ, the production GPU path), then short recipes for the rest. All snippets are starting points; pin your versions and read the linked docs for the details that change.

First, the memory math that predicts everything before you download a single checkpoint:

def weight_memory_gb(params_billion: float, bits: int) -> float:
    return params_billion * bits / 8  # 1e9 params * bits / 8 bits-per-byte / 1e9 bytes-per-GB

for fmt, bits in [("FP32", 32), ("FP16", 16), ("INT8", 8), ("INT4", 4)]:
    print(f"{fmt:>5}: {weight_memory_gb(70, bits):>6.1f} GB")

# FP32: 280.0 GB | FP16: 140.0 GB | INT8: 70.0 GB | INT4: 35.0 GB

Add 10 to 30% on top for KV cache, activations, and CUDA context, scaling with your context length and batch size.

The zero-effort baseline: 4-bit loading with bitsandbytes

Before committing to a full quantization pipeline, sanity-check that a 4-bit version of your model works at all:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,  # quantizes the quantization constants, ~0.4 bits/param saved
)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

This quantizes at load time with zero calibration. Convenient, and good enough quality for many uses, but serving throughput belongs to the dedicated formats below.

Deep walkthrough: AWQ with llm-compressor, served on vLLM

Step 1: quantize with calibration data. This is where most quality is won or lost, so choose calibration samples that resemble your production traffic. Chat model? Use chat data. Code model? Use code. Around 256 to 512 representative samples is the standard range, and representative beats plentiful.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from llmcompressor import oneshot
from llmcompressor.modifiers.awq import AWQModifier

MODEL_ID = "meta-llama/Llama-3.1-8B-Instruct"
NUM_SAMPLES = 256
MAX_SEQ_LEN = 2048
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
# Calibration data: match your production distribution
ds = load_dataset("HuggingFaceH4/ultrachat_200k", split=f"train_sft[:{NUM_SAMPLES}]")
ds = ds.map(lambda ex: {
    "text": tokenizer.apply_chat_template(ex["messages"], tokenize=False)
})
recipe = AWQModifier(
    targets="Linear",
    scheme="W4A16",
    ignore=["lm_head"],  # the output head stays high precision
)
oneshot(
    model=model,
    dataset=ds,
    recipe=recipe,
    max_seq_length=MAX_SEQ_LEN,
    num_calibration_samples=NUM_SAMPLES,
)
model.save_pretrained("Llama-3.1-8B-Instruct-AWQ-W4A16")
tokenizer.save_pretrained("Llama-3.1-8B-Instruct-AWQ-W4A16")

Step 2: serve with vLLM. The compressed-tensors format saved above loads directly:

vllm serve ./Llama-3.1-8B-Instruct-AWQ-W4A16 --max-model-len 8192

Watch the startup logs for the kernel being used (you want to see a Marlin or AWQ kernel mentioned). If vLLM falls back to a generic path, you have compatibility rather than acceleration, and the pitfalls section explains what to check.

Step 3: verify the memory win.

import torch
from vllm import LLM

llm = LLM(model="./Llama-3.1-8B-Instruct-AWQ-W4A16")
print(f"Peak GPU memory: {torch.cuda.max_memory_allocated() / 1e9:.1f} GB")
# Compare against the same measurement on the BF16 checkpoint

Recipe: QLoRA fine-tuning

from transformers import BitsAndBytesConfig
from peft import LoraConfig, prepare_model_for_kbit_training, get_peft_model

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(MODEL_ID, quantization_config=bnb_config, device_map="auto")
model = prepare_model_for_kbit_training(model)
lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj"])
model = get_peft_model(model, lora)
# from here, a normal Trainer / TRL SFTTrainer loop

The frozen base sits in NF4; only the adapters (a fraction of a percent of parameters) train in BF16.

Recipe: GGUF for local inference

# Convert HF checkpoint to GGUF, then quantize to Q4_K_M
python llama.cpp/convert_hf_to_gguf.py ./Llama-3.1-8B-Instruct --outfile model-f16.gguf
./llama.cpp/build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

# Run it
./llama.cpp/build/bin/llama-cli -m model-Q4_K_M.gguf -p "Explain quantization in one paragraph." -n 256

Or skip the ceremony entirely with Ollama, which wraps the same runtime. Note the binary names: older tutorials say quantize and main; the current binaries are llama-quantize, llama-cli, and llama-server.

Recipe: quantized Flux with diffusers

The component-level strategy from the diffusion section, in code. Transformer in 4-bit NF4, T5 in 8-bit, VAE untouched:

import torch
from diffusers import FluxPipeline, FluxTransformer2DModel
from diffusers import BitsAndBytesConfig as DiffusersBnbConfig
from transformers import T5EncoderModel, BitsAndBytesConfig as BnbConfig

MODEL_ID = "black-forest-labs/FLUX.1-dev"
transformer = FluxTransformer2DModel.from_pretrained(
    MODEL_ID, subfolder="transformer",
    quantization_config=DiffusersBnbConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                                           bnb_4bit_compute_dtype=torch.bfloat16),
    torch_dtype=torch.bfloat16,
)
text_encoder_2 = T5EncoderModel.from_pretrained(
    MODEL_ID, subfolder="text_encoder_2",
    quantization_config=BnbConfig(load_in_8bit=True),
    torch_dtype=torch.bfloat16,
)
pipe = FluxPipeline.from_pretrained(
    MODEL_ID, transformer=transformer, text_encoder_2=text_encoder_2,
    torch_dtype=torch.bfloat16,  # VAE and CLIP stay in BF16
)
pipe.enable_model_cpu_offload()
image = pipe("a watercolor fox reading a newspaper", num_inference_steps=28).images[0]

For maximum speed on consumer GPUs, the SVDQuant/Nunchaku engine with prequantized Flux checkpoints is the stronger option; it integrates with ComfyUI and diffusers.

Recipe: FP8 KV cache, one line

from vllm import LLM
llm = LLM(model=MODEL_ID, kv_cache_dtype="fp8")

At long context or high concurrency, the KV cache is often the binding memory constraint rather than the weights. This flag roughly halves it.

Benchmarking: a real case study

Abstract percentages breed distrust, so here is a fully sourced case: Llama 3.1 70B Instruct, BF16 versus INT4 (W4A16, GPTQ algorithm), quantized and evaluated by Neural Magic (now Red Hat AI), served on vLLM.

Source: Red Hat AI model card for Meta-Llama-3.1–70B-Instruct-quantized.w4a16 (full reproduction commands included there). Throughput scales with batch size, serving engine, and GPU; benchmark on your own stack before quoting numbers.

Read this table twice, because both readings matter. Reading one: INT4 cut the deployment in half and the headline averages barely moved; coding actually ticked up within noise. Reading two: look at GPQA, down to 89.9% recovery, and Math-|v|-5 in the full card at 93.5%. Averages hide task-level regressions, and the tasks that regress are usually the hard reasoning ones. If your product depends on exactly those capabilities, the average will lie to you.

Run your own evaluation with lm-evaluation-harness on both checkpoints, and include an instruction-following task:

lm_eval --model vllm \
  --model_args pretrained=./Llama-3.1-8B-Instruct-AWQ-W4A16,dtype=auto \
  --tasks gsm8k,ifeval \
  --batch_size auto
# Repeat with pretrained=<BF16 baseline> and diff the results

Then do the unglamorous thing: run 50 real prompts from your product through both models and read the outputs side by side. Twenty minutes of reading catches failures no leaderboard will.

Production pitfalls

Calibration data mismatch. Static PTQ methods learn their scales from calibration samples, and a few hundred representative samples beat ten thousand random ones. Quantize a code model on Wikipedia text and the activation ranges are simply wrong for your traffic. This is the highest-leverage, least-discussed knob in the whole pipeline. When quality disappoints, check calibration before blaming the method. There is even a subtle failure mode here for instruct models: calibrating on raw text without the chat template underrepresents the special tokens your production traffic always contains.

The perplexity trap. Perplexity is a blunt instrument. Models can lose instruction-following, formatting discipline, and multi-step reasoning while perplexity barely moves, because perplexity averages over all tokens and those capabilities live in a brittle few. IFEval and task-specific evals catch what perplexity misses; the GPQA row in the case study is this pitfall in miniature.

No kernel, no speedup. A quantized model is only fast if a fused low-precision kernel exists for your exact combination of format, GPU architecture, and serving engine. Otherwise you pay dequantization overhead with none of the compute win, and yes, that can be slower than FP16. Check the engine’s startup logs for which kernel was selected, and treat “it loaded fine” as a claim about compatibility rather than performance.

Outlier channels. The recurring villain of this entire field. A handful of activation channels with extreme magnitudes force the quantizer to waste its dynamic range, crushing every normal value. Modern methods are largely a catalog of outlier-handling strategies (AWQ’s channel scaling, SmoothQuant’s migration, SVDQuant’s low-rank absorption). Practical implication: if a model quantizes badly with method A, try a method with different outlier handling before giving up on 4-bit.

Quantizing an already-degraded model. Errors compound. A heavily fine-tuned model sitting at the edge of stability degrades more from quantization than its base model. Evaluate the exact checkpoint you plan to ship.

Shipping without a rollback. Keep the BF16 deployment path warm. Quantization regressions have a habit of surfacing in production on the one workload nobody evaluated.

Where this is heading

Three developments worth tracking, briefly. FP4 and NVFP4 get native tensor core support on Blackwell, and NVIDIA reports near-FP8 quality with proper scaling; expect the FP8 story of 2024 to replay one tier down. QAT is going mainstream: Gemma 3 QAT checkpoints showed that when the model trainer owns quantization, 4-bit quality stops being a compromise, and more labs will ship official low-bit weights. BitNet-style 1.58-bit models remain the wildcard; ternary weights trained from scratch work at small scale, and the open question is whether the recipe holds at frontier scale.

The direction is consistent: quantization is moving from a post-hoc compression trick to a first-class part of how models are trained, released, and served. The engineers who understand the tradeoffs now are the ones whose inference bills look sane later.

If this guide saved you a benchmark cycle or a deployment mistake, that was the goal. The next article in this series covers inference serving engines in the same style. Follow along if that is useful to you.

References

1. Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024. https://arxiv.org/abs/2306.00978
2. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, ICLR 2023. https://arxiv.org/abs/2210.17323
3. Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023. https://arxiv.org/abs/2305.14314
4. Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023. https://arxiv.org/abs/2211.10438
5. Li et al., SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models, ICLR 2025. https://arxiv.org/abs/2411.05007 and the Nunchaku engine: https://github.com/nunchaku-ai/nunchaku
6. Red Hat AI / Neural Magic, Meta-Llama-3.1–70B-Instruct-quantized.w4a16 model card (case study numbers and reproduction commands). https://huggingface.co/RedHatAI/Meta-Llama-3.1-70B-Instruct-quantized.w4a16
7. Kurtic et al., “Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization, 2024. https://arxiv.org/abs/2411.02355
8. Red Hat Developer, LLM Compressor is here: Faster inference with vLLM (memory-bound vs compute-bound serving behavior). https://developers.redhat.com/articles/2024/08/14/llm-compressor-here-faster-inference-vllm
9. AutoAWQ repository deprecation notice (maintainer statement, adoption by the vLLM project). https://github.com/casper-hansen/AutoAWQ
10. llm-compressor (vLLM project). https://github.com/vllm-project/llm-compressor
11. GPTQModel (AutoGPTQ successor). https://github.com/ModelCloud/GPTQModel
12. vLLM documentation: quantization support matrix, FP8, and kv_cache_dtype. https://docs.vllm.ai/en/latest/features/quantization/
13. llama.cpp (GGUF quantization tooling). https://github.com/ggml-org/llama.cpp
14. Hugging Face Diffusers documentation: quantization for Flux and other pipelines. https://huggingface.co/docs/diffusers/quantization/overview
15. Google, Gemma 3 QAT models: bringing state-of-the-art AI to consumer GPUs. https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
16. Dettmers et al., LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale, NeurIPS 2022. https://arxiv.org/abs/2208.07339
17. Ma et al., The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, 2024. https://arxiv.org/abs/2402.17764
18. NVIDIA, Introducing NVFP4 for Efficient and Accurate Low-Precision Inference. https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/