Speeding Up Large Language Models: A Deep Dive into GPTQ and AWQ Quantization

Image by author — Generated with DALL.E 3

This story was written with the assistance of an AI writing program.

Quantization is one of the most powerful techniques available to reduce memory consumption and accelerate inference speed for large language models (LLMs). In this post, I dive into two of the most popular quantization strategies today: GPTQ and AWQ, exploring how they work, how to implement them, and what trade-offs they bring.

What Is Quantization?

Quantization is the process of reducing the precision of the numbers (usually floating-point weights) used in a neural network. For example, instead of using 16-bit or 32-bit floats, quantization allows models to use 8-bit, 4-bit, or even 3-bit integers to represent weights. This significantly reduces the model’s memory footprint and can improve inference speed by allowing more of the model to be cached on faster hardware like GPU memory.

Overview of Popular Quantization Techniques

Two cutting-edge quantization methods that have gained traction for LLMs are:

• GPTQ: A layer-wise post-training quantization method that approximates full-precision model outputs as closely as possible.
• AWQ (Activation-Aware Weight Quantization): Selectively preserves critical weights by considering their impact on activations, using mixed precision (e.g., FP16 for salient weights).

Let’s unpack each.

AWQ: Activation-Aware Weight Quantization

Paper: Activation-Aware Weight Quantization

AWQ’s core insight is this:

Not all weights are equally important. Some weights play a disproportionate role in preserving the model’s predictive performance.

Key Concepts

Illustration of part of AWQ mechanism [Source]

• Salient Weights: A small fraction (less than 1%) of weights have a large impact on the output. These are kept in high precision (e.g., FP16), while the rest are quantized to INT3 or INT4.
• Importance Estimation: Rather than relying on weight magnitude or norms, AWQ uses activation-aware importance defined as:

Importance(w_ij) = E[ |w_ij * a_j| ]

Where:

• w_ij is the weight at row i, column j
• a_j is the activation value at index j
• E[...] denotes the expected value (mean) across inputs

Weights that contribute more strongly to large activations are considered more important and are prioritized during quantization.

• Auto-Scaling: To balance the error of important and unimportant weights, a method is used to automatically search for scaling factors for each input channel. The scaling factors are set based on the input activation size, and the optimal value is quickly found through a simple grid search. It is said that it does not lose generality because it is relatively less affected by the calibration set.
• Experimentally, keeping salient weights in FP16 while quantizing others to INT3 provides excellent accuracy-speed trade-offs.

Implementing AWQ with AutoAWQ

Setup:

• A100 80GB GPU
• PyTorch 2.5.1 with CUDA 12.4 in Docker
• Example model: meta-llama/Llama-3.1–8B-Instruct

pip install autoawq

If you’re using multi-GPU for large models (e.g., LLaMA 3.1 70B), you might face:

RuntimeError: expected all tensors to be on the same device

Fix:

pip uninstall transformers # Use patched transformers
git clone https://github.com/davedgd/transformers -b patch-1
cd transformers && python setup.py install

Next:

git clone https://github.com/casper-hansen/AutoAWQ.git
cd AutoAWQ/examples
python quantize.py # model_path /path/to/llama3.1-8b, quant_path /path/to/output

Only 4-bit quantization is currently supported. Choose backend via version: marlin, gemm, gemv, etc.

• gemv = best for batch size 1
• gemm = optimized for long contexts

Enable multi-GPU like below:

model = AutoAWQForCausalLM.from_pretrained(model_path, low_cpu_mem_usage=True, use_cache=False, device_map="auto")

The AWQ process quantizes all 32 layers of the model and typically takes around 10 minutes to complete. During this process, the GPU memory usage can peak at approximately 24GB. This memory requirement arises because the full model must be loaded into memory, which means the memory needed often exceeds the model’s weight size (about 16GB). Additionally, some extra memory is consumed during the quantization steps themselves, due to temporary buffers and computations.

Calib data (default): mit-han-lab/pile-val-backup (Other data is also possible)

Output:

Quantized files include model.safetensors, quantize_config.json, model.safetensors.index.json, and updated config.json with:

"quantization_config": {"bits": 4, "group_size": 128, ...}

GPTQ: Optimal Brain Quantization

Paper: GPTQ: Accurate Post-Training Quantization for Generative Transformers

GPTQ focuses on layer-wise quantization, approximating the full-precision outputs of each layer using quantized weights.

Key Concepts

Figure of GPTQ quantization procedure [Source]

• Layer-Wise Quantization: Quantize each layer independently to reduce cumulative error.
• Optimal Brain Quantization (OBQ): Sequentially quantize weights in each row and compensate for quantization error in future weights via Hessian updates.
• Cholesky Reformulation: Ensures numerically stable and efficient Hessian matrix updates.

GPTQ Improvements

1. Arbitrary Order: All weights treated equally to avoid redundant sorting.
2. Lazy Batch Updates: Update Hessian inverses in blocks to reduce overhead.
3. Efficient Hessian Updates: Faster convergence and lower complexity using Cholesky reformulation.

Implementing with AutoGPTQ

Setup:

• A100 80GB GPU
• PyTorch 2.5.1 with CUDA 12.4 in Docker

Because CUDA version 12.1 or higher is required, a simple pip install auto-gptq --no-build-isolation is required according to guide. However, it may not be sufficient, and can result in a cuda extension not installed error. To resolve this, it's necessary to install AutoGPTQ from source as follows:

git clone https://github.com/AutoGPTQ/AutoGPTQ.git
cd AutoGPTQ
pip install numpy gekko pandas
pip install -vvv --no-build-isolation -e .

Usage:

cd examples/quantization
python3 basic_usage_wikitext2.py # pretrained_model_dir /path/to/llama3.1-8b-instruc, quantized_model_dir /path/to/output

Calib dataset: Wikitext2 (default) or replace with Alpaca via quant_with_alpaca.py.

You may replace the calibration dataset with another Hugging Face dataset. Calibration dataset choice can influence quantization quality, though the exact extent may require further experimentation.

Supports 2, 3, 4, or 8-bit quantization. You can configure quantization settings like bits, group_size, or desc_act in the BaseQuantizeConfigaccording to this.

If you want to use the Marlin kernel (an extremely optimized FP16xINT4 matmul kernel for LLM inference), set:

quantize_config = BaseQuantizeConfig(checkpoint_format="marlin", ...)

Refer to: https://github.com/IST-DASLab/marlin for more information.

During quantization, the model is processed layer-by-layer. While the model (around 16GB in FP16) is fully loaded, GPU usage remains moderate:

• Max: 7.5GB
• Avg: 4MB

Because the quantization operates on one layer at a time, the overall GPU memory consumption is lower than AWQ, though the quantization duration is longer.

Takes ~25 minutes to quantize a 32-layer model.

Output includes:

• gptq_model-4bit-128g.safetensors
• quantize_config.json
• config.json (updated with "checkpoint_format": "gptq")

Serving Quantized Models with vLLM

vLLM supports serving AWQ or GPTQ models via the --quantization flag.

Quantization reduces memory use, allowing for larger kv_cache, higher concurrency, and faster inference speeds.

Why Quantized Models Are Faster

Quantization significantly speeds up inference by reducing the computational and memory burdens on hardware. Since quantized weights use fewer bits, they require less memory bandwidth and storage, leading to lower latency and higher throughput.

Memory-Bound vs Compute-Bound

• Memory-Bound (e.g., small batch sizes): Faster with quantization because weights are smaller and memory bandwidth is the bottleneck.
• Compute-Bound (e.g., large batch sizes): Speed-up diminishes due to dequantization work (INT4-to-FP16 conversion overhead).

In compute-bound scenarios, activation quantization (not just weights) may be needed for real speed-up.

Reference: Red Hat — LLM Compressor

Conclusion

Quantization is an essential technique to scale LLMs efficiently.

• AWQ prioritizes important weights based on activation influence.
• GPTQ minimizes output error via layer-wise Hessian-based optimization.

Whether you are deploying on resource-constrained edge devices or trying to serve high QPS workloads, quantization will be a critical part of your LLM optimization toolkit.

Stay tuned for future benchmarks comparing AWQ vs GPTQ on real-world inference workloads!

If you found this guide helpful, feel free to ask any questions in the comments. Don’t forget to hit the like button and subscribe for more content! 😊