Use Quantization to maximize LLM experience

The advent of Large Language Models (LLMs) has spurred efforts to enhance performance while reducing computational demands, resulting in the creation of diverse quantization techniques.

Quantization is a method that lowers the precision of numeric values within a model. It employs lower-precision data types, like 8-bit integers, rather than high-precision types such as 32-bit floating points. This technique substantially cuts down on memory consumption and can accelerate the operation of models, all while preserving a satisfactory level of accuracy.

Unless you are working on small models or you don’t need worry about GPU memory size, you will need consider to adopt quantization method for model training or inference sooner or later. Even further, a better understanding of the quantization method will help you to shorter the training time, and make the model inference more efficient.

What exactly does Quantization mean?

Quantization is fundamentally straightforward: it involves transitioning from a higher precision format, typically the standard 32-bit floating-point, to a more compact, lower precision data type for storing weights and activations within a model. The most commonly adopted lower precision data types include:

• float16, with an accumulation data type of float16
• bfloat16, with an accumulation data type of float32
• int16, with an accumulation data type of int32
• int8, with an accumulation data type of int32

To illustrate, consider two int8 values: A = 127 and B = 127. If we compute the sum C = A + B, the result exceeds the maximum value storable in an int8 data type. Thus, without using a higher precision data type for accumulation, the sum would lead to substantial precision loss. This necessity for an increased precision during accumulation steps ensures that the quantization procedure remains practical and efficient.

Quantization Types

There are two types of quantization:

• Post-Training Quantization: Apply quantization to a trained model to decreased memory usage, quicker computation during inference, and heightened energy efficiency. Nonetheless, this conversion can lead to a decrease in the model’s accuracy, as there is typically a disparity between the behavior of the original full-precision model and the performance of its quantized version.
• Quantization-Aware model: Integrate quantization directly into the model training pipeline to refine the model’s weights and minimize the accuracy degradation. Since the model is calibrated during the training itself, the model will be better maintain its accuracy.

(Source: A Survey of Quantization Methods for Efficient Neural Network Inference)

Quantization Models and Libraries

As described above, there are some models are trained quantized, and becomes to quantization models. Normally, people like uses prefix to append their model names on what quantization method the model being trained. For example, the TheBloke publishes their quantized models in Hugging Face as following.

When choose a quantized model, you need read the model card on carefully on prompt template, compatibilities, model parameters, etc.

Those are very important when you inference the model, or fine tune the model with your own data. Mis-configurations might lead to broken experience or unbelievable long process time for training or inference.

When choosing quantized models, you need also adopt the same quantization libraries. Below are popular quantization methods (library and relevant model) being used today:

• GGML: GGML is a optimized for tensor operations and supports different quantization formats to reduce memory and computational costs. GGML is tailored for both Apple M1/M2 and x86 architectures, leveraging AVX/AVX2 instructions. Developed by Georgi Gerganov, GGML stands out for having no dependencies and not allocating memory at runtime, facilitating easy integration and deployment for on-device ML inference. For more examples on how to use GGML, please check out below article.

[embed]LLM by Examples — Use GGML Quantization What is GGML?medium.com

• GGUF: GGUF is an evolved framework enhancing GGML for running Large Language Models efficiently on CPUs, with optional GPU support for specific layers, and optimizes model loading and storage with compressed weights. For more practical examples, please refer to below article.

[embed]LLM By Examples — Use GGUF Quantization What is GGUF?medium.com

• GPTQ: GPTQ, which stands for Generalized Post-Training Quantization, is a technique tailored for the quantization of neural network weights to a 4-bit representation primarily to enhance performance on GPU-based inference tasks. The core principle of GPTQ is to condense the model’s weights into a 4-bit format by aiming to reduce the mean squared error between the original weights and their quantized counterparts. This process of quantization is done after the model has been trained, optimizing the network to maintain its accuracy despite the significant reduction in data precision. To get quick started, please check examples from below link:

[embed]LLM By Examples — Use GPTQ Quantization GPTQ is a technique for compressing deep learning model weights through a 4-bit quantization process that targets…medium.com

• LoRA and QLoRA: LoRA is a fine-tuning method that updates low-rank, small matrices within a pre-trained model to efficiently achieve task-specific adaptation. QLoRA combines LoRA’s methodology with 4-bit weight quantization, optimizing efficiency and maintaining model effectiveness during fine-tuning. Below article gives deep dives as well as code examples on how to fine tune models using LoRA and QLoRA.

[embed]LLM By Examples — Using LoRA and QLoRA Within the field of large language model (LLM), it’s typical to tailor a pre-trained model for a particular function by…medium.com

• Bitsandbytes and NF4 allows you to load bigger model in small GPU memory, and be able to offload CPU memory. The biggest benefits are easy-to-use, and small GPU memory footprint, and trade-off is the speed during inference. Below article shows a few examples on how to use bitsandbytes for quantization:

[embed]LLM By Examples — Use Bitsandbytes for Quantization What is Bitsandbytes?medium.com

Besides choosing quantized models, you could also apply quantization to regular models with above quantization approaches during fine tuning. Just bear in mind:

• The compression time for a not-quantized model at first time is very time consuming
• Your fine tuning dataset might impact the efficiency of quantization. Always try to start with data from base model during inference.

Inference vs. Fine Tuning

Below decision chart gives you a general idea on how or when to consider quantization.

Have Fun!