LLM Quantization, Distillation, and Optimization Guide

A large language model can perform brilliantly in a benchmark and still be impractical in production. It may require more memory than the available hardware, respond too slowly for an interactive product, or become expensive once thousands of users begin sending requests.

Quantization, distillation, and optimization address those problems in different ways:

· Quantization reduces the numerical precision used to store or process the model.

· Distillation trains a smaller model to reproduce useful behavior from a larger one.

· Optimization improves how the model is trained, loaded, scheduled, and executed.

These methods are complementary. A distilled model can be quantized, then deployed through an optimized inference engine with efficient batching and cache management.

How the Techniques Differ

Where you start depends on what the main bottleneck is.

If your model does not fit in memory, try quantization first. If it fits but still needs too much computation, you might need a smaller or distilled model. If performance drops when handling many requests at once, focus on batching, scheduling, cache management, or request routing.

Quantization: Storing and Processing Smaller Numbers

A language model is mostly made up of numerical parameters. For example, a model with seven billion parameters stores about seven billion values, along with activations, runtime buffers, attention data, and cached keys and values.

The precision used for each parameter has a major effect on storage. A 32-bit value occupies four bytes, a 16-bit value two bytes, an 8-bit value one byte, and a 4-bit value half a byte.

Approximate weight storage looks like this:

These figures cover the weights alone. An actual deployment also needs memory for the KV cache, temporary workspaces, quantization scales, model metadata, and the inference engine. A model described as “4-bit” will therefore use more memory than the raw weight calculation suggests.

How Quantization Works

Quantization takes a wide range of high-precision values and converts them into a smaller set of low-precision values. Rather than saving each original floating-point number, the system keeps an integer or low-precision version and some scaling information.

A basic way to calculate this is:

quantized value = round(original value ÷ scale) + zero point

The scale can be worked out for a whole tensor, a single channel, or a small group of weights. Using smaller groups usually keeps accuracy higher because they can adjust to local value patterns. However, this approach needs extra metadata and sometimes makes execution more complex.

Weight-Only Quantization

Weight-only quantization reduces the precision of model weights but keeps activations at a higher precision, usually FP16 or BF16. For example, the W4A16 format uses 4-bit weights and 16-bit activations.

This method is popular because model weights take up most of the static memory and are read many times during generation. Making them smaller can reduce memory bandwidth demands, especially when decoding tokens one at a time.

However, a 4-bit model does not always use 4-bit arithmetic for every calculation. Some runtimes decompress the weights before multiplying, while others use special low-bit kernels. This detail affects whether quantization actually speeds things up or just saves storage.

Weight-and-Activation Quantization

Formats like W8A8 quantize both weights and activations. This helps lower memory traffic and lets more operations use efficient low-precision matrix multiplication.

Activations are harder to quantize than weights because their values depend on the input and may contain unusually large outliers. SmoothQuant addresses this problem by shifting some of the quantization difficulty from activations into weights through a mathematically equivalent scaling transformation.[1]

Modern accelerators may also support FP8, an 8-bit floating-point format with a wider dynamic range than conventional 8-bit integers. FP8 can be useful for both training and inference, but the benefit depends heavily on native hardware and software support.

KV-Cache Quantization

During autoregressive generation, a Transformer stores key and value tensors for previously processed tokens in a KV cache. This prevents the model from recalculating the entire conversation whenever it generates another token.

The cache grows with sequence length, batch size, number of layers, and number of key-value heads. In long-context or high-concurrency systems, it can consume as much memory as the model weights — or more.

Quantizing the KV cache helps support longer contexts and bigger batch sizes. However, if the cache is quantized too much, it can reduce attention accuracy and output quality, especially for long or complex tasks.

Post-Training Quantization

Post-training quantization (PTQ) compresses a model after it has already been trained. Most of the time, you do not need to retrain the model, but some methods use a small calibration dataset to check typical weight and activation patterns.

PTQ is popular because it is much faster than retraining. Teams can start with a full-precision model, calibrate it with realistic prompts, create a quantized checkpoint, and test the results without having to train the model again.

Common approaches include:

GPTQ uses approximate second-order information to reduce the error introduced when weights are quantized. It was designed as a one-shot method for compressing large generative models, including at 3-bit and 4-bit precision.[2]

AWQ, or Activation-Aware Weight Quantization, uses activation statistics to identify particularly important weight channels. It applies scaling to protect those channels instead of relying on costly mixed-precision execution.[3]

SmoothQuant focuses on activation outliers and is closely associated with practical W8A8 deployment.[1]

These methods are not interchangeable. Their quality and performance depend on the model architecture, group size, calibration data, quantization format, runtime, kernels, and target hardware.

Quantization-Aware Training

Quantization-aware training, or QAT, mimics low-precision behavior while the model is being trained. This way, the model learns to handle rounding and clipping errors as it trains, instead of facing them only after training is finished.

QAT can preserve more quality at aggressive bit widths, but it is more expensive than PTQ. It requires suitable training data, additional optimization, and careful handling of numerical stability.

It is most attractive when a model will be deployed at large scale, when a specific low-bit format is mandatory, or when post-training quantization causes unacceptable quality loss.

QLoRA Is Related, but Not the Same

QLoRA combines a frozen, 4-bit quantized base model with trainable low-rank adapters. Gradients pass through the quantized model, but only the adapter parameters are updated. The method introduced NormalFloat 4-bit quantization, double quantization, and paged optimizers to reduce the memory required for fine-tuning.[4]

QLoRA is primarily a fine-tuning technique, not simply an inference format. After training, the adapters may remain separate or be merged into a deployment checkpoint, depending on the serving system.

Why Smaller Models Are Not Always Faster

Quantization consistently lowers the amount of storage needed for weights. However, its impact on speed can vary.

A model with fewer bits might run slowly if the hardware does not have efficient support for its format. The extra work needed to dequantize can offset the advantage of reading less data. Other factors, like short prompts, small batch sizes, CPU to GPU transfers, or system bottlenecks, can also reduce the expected speedup.

Because of this, it is better to judge a quantization format by actual measurements instead of just file size. Some useful metrics are:

· Peak memory consumption

· Time to first token

· Time per output token

· Tokens generated per second

· Throughput under realistic concurrency

· Quality on application-specific tasks

· Performance at typical and maximum context lengths

Starting with eight-bit quantization is usually a safe choice. Using four bits can save more memory, but it needs more careful testing. If you go below four bits, the results depend a lot on the model, calibration data, quantization method, and the specific workload.

Distillation: Training a Smaller Student Model

Quantization compresses the numerical representation of an existing network. Distillation goes further by training a smaller student model to learn from a larger teacher.

Traditional knowledge distillation involves more than just using correct labels. The student also learns from the teacher’s probability distribution. These “soft targets” show connections that hard labels miss. For example, a teacher might strongly prefer one answer, see another as possible, and give very low probability to the rest. This approach gives the student better guidance.[5]

For generative models, distillation is more complicated because the output is a sequence rather than a single label. Each generated token changes the context for the next one, and many different responses may be valid.

Black-Box and White-Box Distillation

In black-box distillation, the student has access only to the teacher’s generated responses. Prompts are sent to the teacher, and the resulting prompt-response pairs become training data.

This method works well when the teacher can only be accessed through an API. It allows you to transfer things like task behavior, formatting rules, writing style, and patterns specific to the field. Still, it leaves out most of the information found in the teacher’s token probabilities and internal data.

In white-box distillation, the training process can also use logits, token probabilities, hidden states, attention relationships, or intermediate features. This provides a richer learning signal but requires direct access to the teacher model.

Methods such as MiniLM have shown that a student can learn by reproducing relationships inside the teacher’s self-attention mechanism rather than copying only its final predictions.[6]

Response Distillation

The simplest form of LLM distillation is sequence-level imitation:

1. Build a representative set of prompts.

2. Ask the teacher to answer them.

3. Filter, score, or verify the answers.

4. Fine-tune the student on the resulting examples.

The dataset must resemble the conditions in which the student will be used. Training only on polished, straightforward prompts can produce a model that struggles with ambiguity, malformed inputs, adversarial requests, rare cases, or incomplete information.

Teacher-generated data should be treated as unverified training material, not ground truth. Large quantities of weak supervision do not become reliable simply because they are large.

Logit and Token-Level Distillation

When the teacher’s output distribution is available, the student can learn from token probabilities rather than only the selected response.

This is useful because a generated answer hides the alternatives the teacher considered. Token-level supervision preserves more of that information.

Some LLM distillation methods also train the student on sequences it generates itself. This reduces the gap between training and deployment. A student trained only on perfect teacher responses may struggle after making an imperfect prediction, because that prediction changes the context for everything that follows.

MiniLLM, for example, used reverse-KL-based objectives and student-generated samples to improve generative distillation for smaller language models.[7]

Distilling Explanations and Intermediate Supervision

A teacher does more than simply provide the final answer. They can share worked examples, explain ideas clearly, demonstrate tools, offer feedback, label different parts of a problem, and show each step in the process.

Research such as Distilling Step-by-Step found that when teachers explain their reasoning, smaller models learn faster and do better on tasks.[8] This does not mean a small model will always perform as well as a larger one. However, it does show that detailed guidance helps transfer problem-solving skills more effectively than just giving final answers.

A distilled model can get very good at tasks such as invoice extraction, customer support routing, document classification, or following a certain reasoning style. Still, it may have trouble with unfamiliar topics, rare languages, subtle conversations, or complex open-ended problems.

What Distillation Can Achieve

A distilled student model might have fewer layers, a smaller hidden size, fewer attention heads, or a reduced vocabulary and embedding setup. This means the model does less work for each token.

Unlike weight-only quantization, distillation can reduce both storage and computation. A compact student may also be easier to deploy on CPUs, mobile devices, edge hardware, or low-cost accelerators.

Distillation works especially well when the target task is narrower than the teacher’s full range of capabilities. There is little value in paying for a large general-purpose model when an application needs to classify documents into a small number of categories.

What Can Go Wrong

A student model does not always learn all of the teacher model’s abilities equally well.

Some rare skills might be lost if they do not appear often in the training data. Longer responses can lose quality more than shorter ones. Multilingual abilities may become focused on the most common language in the data. Safety features can also weaken if there are not enough examples of refusals or challenging prompts.

The student model can also pick up the teacher’s mistakes, biases, and writing style. Because of this, synthetic data should be filtered, checked for duplicates, scored, and tested. You can verify factual outputs by comparing them to databases, calculators, code, retrieval tools, or by having a person review them.

Licensing and platform terms also matter. Before using teacher outputs to train or redistribute a student model, teams should confirm that the teacher’s license or API terms permit the intended use.

A Practical Distillation Workflow

A successful distillation project starts by setting a clear deployment target, rather than just aiming to make the model smaller.

Decide on the application, how much quality loss is acceptable, latency goals, memory limits, supported languages, and safety needs. Pick the student model architecture based on the hardware and workload you plan to use.

Create your training set using real or realistic data. Make sure to include typical requests, tough cases, malformed inputs, situations where the model should refuse, long contexts, and examples outside the main domain. If you can, generate multiple teacher responses and check or rank them before training.

Use both reliable human data and synthetic teacher data. If you have white-box access, combine hard targets with logits or objectives at the representation level. In the end, test the student model on the main task as well as on broader regression tests.

If the student model gets better at extraction but can no longer follow instructions or stay safe, that is not a real improvement for production.

Optimization: Making the System Waste Less Work

Optimization covers more than model compression. It includes kernels, memory allocation, caching, batching, scheduling, parallelism, prompt design, request routing, and application logic.

For inference, it helps to separate two stages: prefill and decode.

During prefill, the model processes the input prompt and builds the KV cache. Prompt tokens can be processed largely in parallel, so this stage is often compute-intensive.

When decoding, the model creates tokens one by one. At each step, it reads the model weights and cached attention data again. This process is often slowed down by memory bandwidth and low hardware use.

An improvement that reduces time to first token may therefore have little effect on the speed of later tokens, and vice versa.

Process Fewer Tokens

The most direct optimization is to remove unnecessary tokens.

Long prompts, repeated instructions, large documents, wordy tool definitions, and lengthy outputs all add to computation. Streamlining these can lower latency and costs without changing the model.

Some helpful changes are:

· Removing repeated or contradictory instructions

· Retrieving fewer, more relevant document passages

· Summarizing older conversation history

· Setting realistic output limits

· Using concise structured templates

· Sending routine tasks to smaller models

· Caching safe, repeatable results

It’s important to balance token reduction with answer quality. If you remove helpful context, you might get worse responses, need more retries, or require human help, which could end up costing more than the original prompt.

Reuse Shared Prefixes

Many requests start with the same system prompt, policy document, reference text, or a few example inputs. Prefix caching saves the KV-cache blocks from this shared start and uses them again in later requests.

This reduces repeated prefill work when prompts share the same prefix. It does not normally accelerate the generation of new output tokens, so it is most valuable when shared prompts are long and frequently reused.

Use Efficient Attention and Fused Kernels

Attention performance is shaped not only by arithmetic but also by data movement.

FlashAttention reorganizes exact attention computation so intermediate values are tiled and reused in fast on-chip memory rather than repeatedly transferred to and from slower high-bandwidth memory. It reduces memory traffic without approximating the attention result.[9]

Fused operations, optimized matrix-multiplication libraries, compiler specialization, graph capture, and architecture-specific kernels all aim to do the same thing: they help reduce memory transfers, kernel launches, and coordination overhead.

Two runtimes can perform the same mathematical operations and still have very different latency because one uses the hardware far more efficiently.

Manage the KV Cache Efficiently

KV-cache allocations change as requests arrive, generate different numbers of tokens, and finish at different times. Naive allocation can waste memory through fragmentation or reserve more space than a request ultimately needs.

PagedAttention applies a paging approach to KV-cache storage. The technique, used by vLLM, allocates and shares cache blocks more efficiently across requests.[10]

Better cache management allows a server to handle longer contexts or more simultaneous requests before exhausting memory. Under production load, this can matter more than small improvements to matrix multiplication.

Use Continuous Batching

Traditional static batching waits for a group of requests, processes them together, and keeps the batch intact until every request finishes. That is inefficient for text generation because response lengths vary widely.

Continuous batching allows completed sequences to leave and new requests to join between decoding steps. This keeps the accelerator busy instead of forcing short requests to wait for the longest response in the batch.

Modern inference engines usually use continuous batching, paged cache management, chunked prefill, optimized kernels, and request scheduling together.

Using larger batches often boosts overall throughput, but it can also lead to longer queueing delays and higher latency for each request. The ideal setup depends on whether the product needs more capacity, faster response times, or strict limits on worst-case latency.

Apply Speculative Decoding Selectively

Autoregressive generation normally requires one large-model pass for every new token.

Speculative decoding uses a smaller draft model to propose several tokens. The larger target model verifies those proposals in parallel. When the acceptance procedure is implemented correctly, the canonical method preserves the target model’s output distribution rather than merely producing approximate text.[11]

The method works best when the draft model is much faster, and its proposed tokens are frequently accepted. A weak, oversized, or poorly matched draft model can add work without improving latency.

Speculative decoding is therefore useful for selected memory-bound and latency-sensitive workloads, not as a universal speed setting.

Choose an Efficient Attention Architecture

Model architecture affects deployment efficiency before runtime optimization begins.

Multi-query attention shares one set of key-value heads across all query heads, sharply reducing KV-cache size and memory traffic. Grouped-query attention uses several groups of shared key-value heads, providing a compromise between conventional multi-head attention and multi-query attention.

Research on grouped-query attention found that it could retain much of the quality of multi-head attention while approaching the decoding efficiency of multi-query attention.[12]

These are architectural choices. They cannot always be added to an existing checkpoint without conversion, adaptation, or additional training.

Use Pruning Carefully

Pruning removes weights or structural components that contribute relatively little to the model’s output.

Unstructured pruning sets individual weights to zero. It can create high sparsity, but ordinary dense hardware may continue processing those zero values unless specialized sparse kernels are available.

Structured pruning removes entire channels, neurons, attention heads, or layers. It is easier for standard hardware to exploit, but each removal affects a larger part of the network and may cause more noticeable quality loss.

Methods such as SparseGPT and Wanda have shown that large language models can be pruned substantially with limited degradation under certain evaluation conditions.[13][14] Those results do not guarantee proportional speed gains.

A model that is 50 percent sparse is not automatically twice as fast.

Optimize Fine-Tuning

Efficiency work also applies during training and adaptation.

Mixed-precision training reduces memory use and can improve accelerator utilization. Gradient checkpointing saves memory by recomputing selected activations during the backward pass. Distributed optimizers partition parameters, gradients, and optimizer states across devices.

LoRA freezes the original model and trains small low-rank update matrices, greatly reducing the number of trainable parameters.[15] QLoRA combines this approach with a quantized backbone to lower the memory required for fine-tuning even further.[4]

These methods help lower the cost of adaptation, but they do not always make the base architecture smaller. You still need the original model for inference unless you compress, distill, or replace it separately.

Combining Quantization, Distillation, and Optimization

The strongest deployments usually combine several techniques.

A distilled model can be quantized. The quantized model can run inside an optimized serving engine with continuous batching, prefix caching, efficient attention kernels, and paged KV-cache management. The application can then route simple requests to smaller models and reserve larger models for difficult cases.

A practical sequence depends on the deployment goal:

Many teams find this order to be the safest approach:

1. Start by measuring quality and performance using full precision.

2. Next, switch to an optimized inference runtime.

3. Adjust batching, cache allocation, prompt length, and output limits as needed.

4. Try moderate quantization to see its effects.

5. Only use lower bit widths if you really need to.

6. If the original model is still too large or complex, consider distilling a smaller version.

Distillation usually needs more data and testing than quantization, so it is not often the first step unless your task is already narrow and clearly defined.

Benchmarking LLM Efficiency

A benchmark should resemble the real application, not an idealized demonstration.

Choose prompt lengths, output lengths, concurrency levels, languages, retrieval context, tool calls, and sampling settings that reflect real use cases. Let the system warm up before you start measuring. When you compare different setups, make sure the model, tokenizer, prompt, generation settings, and stopping rules all stay the same.

Track more than one performance metric.

Time to first token measures how long the user waits before the response begins. It is strongly affected by prompt length, queueing, and prefill performance.

Time per output token measures the speed and smoothness of generation after the first token.

End-to-end latency captures the complete user experience.

Throughput measures requests or tokens processed per second. Batch size and concurrency should always be reported alongside it.

Tail latency, such as P95 or P99, shows what slower users experience under load.

Peak memory use determines whether the model can run and how many requests the system can serve simultaneously.

Quality should be evaluated on the actual application, including factual accuracy, instruction-following, formatting, safety, robustness, and edge cases.

Looking at the cost for each successful task is often more helpful than just checking the cost per token. A cheaper model that makes more mistakes, needs more retries, or requires human help can end up costing more overall.

Throughput and latency usually work against each other. If you batch requests aggressively, you can process more tokens per second, but each request might take longer to complete. When tuning for production, it is important to balance hardware use with how quickly users get responses.

Final Perspective

Quantization, distillation, and optimization solve different parts of the efficiency problem.

Quantization changes numerical precision. It is often the fastest route to lower memory use and can improve inference speed when the format is well supported by the hardware and runtime.

Distillation changes the model. It trades training effort and some breadth of capability for a smaller network that performs less work per token.

Optimization changes execution. It improves how prompts, kernels, caches, batches, devices, and requests work together.

The most efficient language model is not necessarily the one with the fewest parameters or the lowest bit width. It is the system that meets its quality target with the least memory, latency, energy, and operational cost under the workload it will actually face.

References

1. Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.

2. Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-Trained Transformers.

3. Lin et al., AWQ: Activation-Aware Weight Quantization for LLM Compression and Acceleration.

4. Dettmers et al., QLoRA: Efficient Fine-tuning of Quantized LLMs.

5. Hinton, Vinyals, and Dean, Distilling the Knowledge in a Neural Network.

6. Wang et al., MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers.

7. Gu et al., MiniLLM: Knowledge Distillation of Large Language Models.

8. Hsieh et al., Distilling Step-by-Step: Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes.

9. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness.

10. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention.

11. Leviathan, Kalman, and Matias, Fast Inference from Transformers via Speculative Decoding.

12. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints.

13. Frantar and Alistarh, SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot.

14. Sun et al., A Simple and Effective Pruning Approach for Large Language Models.

15. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models.