Comparative Study of Quantized and Parameter-Efficient Fine-Tuning MethodAbstract

Fine-tuning large language models is computationally expensive because standard full fine-tuning requires updating and storing all model parameters. Parameter-efficient fine-tuning (PEFT) methods reduce this load by freezing most pretrained weights and learning a subset of the full weight matrix (W) [1].

This paper compares several PEFT methods such as LoRA [1], QLoRA [2], AdaLoRA [3], and a DyLoRA-style implementation [4] under a limited-scale experimental setup. The experiments are performed using TinyLlama-1.1B-Chat-v1.0 and the yahma/alpaca-cleaned dataset for instruction tuning.

Evaluation is conducted using both downstream-task perplexity on the Alpaca validation split and general language perplexity on the WikiText-2 benchmark. GPU memory usage and training runtime are also analyzed to compare efficiency trade-offs between the methods.

The results show that LoRA achieves the best overall perplexity and the most consistent performance among all evaluated methods. QLoRA provides the best trade-off between performance and memory efficiency by reducing GPU memory usage and maintaining competitive perplexity values. The DyLoRA-style implementation also shows competitive behavior under reduced memory conditions. In contrast, AdaLoRA shows weaker performance under the current setup, due to additional overhead introduced by adaptive rank allocation.

The study highlights that different PEFT methods optimize different objectives, so the best method depends on the target constraint, whether it is model quality, memory efficiency, or computational cost.

Introduction

Large language models (LLMs) have achieved strong performance in many tasks such as text generation, classification, and reasoning. These models are usually pretrained on large datasets and then adapted to downstream tasks using fine-tuning.

The standard approach is full fine-tuning, where all model parameters are updated. However, this requires very high memory and computational resources, especially for large models.

To avoid updating all model parameters, parameter-efficient fine-tuning (PEFT) methods were introduced [5]. The main idea is that it is not necessary to update all parameters of the model. Instead, only a small subset of parameters is trained, or small additional components are added, while keeping most of the pretrained model frozen. This makes training more efficient while still achieving strong performance.

Among PEFT methods, LoRA (Low-Rank Adaptation) is one of the most widely used approaches [1]. It assumes that model updates can be represented in a low-rank subspace, allowing the model to learn task-specific changes using a small number of parameters.

Several variants and extensions have been proposed, such as:

• QLoRA [2]
• AdaLoRA [3]
• DyLoRA [4]

Each improves different aspects such as efficiency, flexibility, or memory usage.

QLoRA extends LoRA by combining quantization mechanisms [2]. The base model is quantized to 4-bit precision, while LoRA adapters are trained on top of it. This helps make large models functional on limited hardware.

Although fixed-rank methods such as LoRA and QLoRA are effective, they assume that the same rank budget is suitable across targeted layers. Adaptive-rank methods challenge this assumption.

AdaLoRA extends LoRA by dynamically allocating rank across layers based on importance of the dimension, instead of using a fixed rank [3]. This leads to better performance under a fixed parameter budget.

DyLoRA removes the need for manual rank selection by enabling dynamic rank adaptation during training using a method different from AdaLoRA [4].

This study focuses on understanding the trade-offs between fixed-rank and adaptive-rank PEFT methods when combined with low-bit quantization.

In practice, full fine-tuning is expensive because it requires updating all model parameters, which leads to high memory usage. Therefore, this work evaluates several PEFT methods under practical limited-scale experimental setup using Google Colab GPU environments.

The paper addresses three research questions:

RQ1

How do adaptive-rank and dynamic-rank methods behave under practical limited-scale experimental conditions?

RQ2

How much GPU memory reduction can quantized PEFT methods such as QLoRA and DyLoRA-style implementations achieve compared to standard LoRA?

RQ3

Does reducing GPU memory usage through quantization necessarily lead to shorter training time?

Related Work

The broader PEFT literature aims to make pretrained language model adaptation practical by reducing the number of trainable parameters.

Early adapter-based work inserted small trainable modules into transformer layers while freezing the pretrained backbone, showing that task-specific transfer does not require updating all model weights [8].

LoRA is based on the observation that the update required during adaptation may exist in a low-dimensional subspace [1]. Instead of directly updating a full weight matrix, LoRA freezes the pretrained matrix (W) and learns a low-rank update represented by two smaller matrices (A) and (B).

The LoRA update is represented as: W_{LoRA} = W_0 + BA

This reduces the number of trainable parameters and can also avoid additional inference latency when the update is merged into the base weights.

Because LoRA is simple, effective, and compatible with modern transformer architectures, it has become a common baseline for efficient fine-tuning.

QLoRA extends LoRA by combining it with low-bit quantization. Dettmers et al. (2023) show that large models can be fine-tuned by loading the pretrained weight matrix in 4-bit precision while training LoRA low-rank matrices.

QLoRA introduced practical mechanisms such as:

• NormalFloat 4-bit quantization
• Double quantization

These mechanisms reduce memory consumption and make it possible to fine-tune larger models with relatively low computational power.

AdaLoRA attempts to solve a limitation of LoRA, which is the fixed rank over the entire training phase [3]. AdaLoRA introduces an adaptive budget allocation strategy in which rank components are scored by importance and less important ones are pruned over time.

DyLoRA, proposed by Valipour et al. (2023), focuses on dynamic rank selection by training LoRA adapters in a way that allows using different rank sizes without retraining the whole model.

In this paper, the original DyLoRA algorithm was not fully reproduced since there is no standard official implementation like AdaLoRA. Instead, a DyLoRA approximation was used where a random active rank is sampled during each training step.

This is considered an experimental technique and not an exact implementation of the original DyLoRA method.

Methodology

Model and Dataset

The base model used in all experiments is:

• TinyLlama/TinyLlama-1.1B-Chat-v1.0 [9]

This model is selected because it is small enough to be fine-tuned on Google Colab GPU environments while still being a transformer-based chat model with more than 1B parameters.

The yahma/alpaca-cleaned dataset is used for fine-tuning because it provides an instruction-following format.

Each example is formatted using:

• instruction
• optional input
• response

The experiment uses:

• 3000 training samples
• 500 validation samples

The dataset is shuffled using seed 2026, while the training run uses seed 42.

The maximum token length is set to 512.

For evaluation, two studies are performed:

In-domain Evaluation

The models are evaluated on the Alpaca validation split.

Out-of-domain Evaluation

The models are evaluated on the WikiText-2 benchmark using sliding-window perplexity.

All methods are trained for:

• 300 steps
• learning rate (2 \times 10^{-4})
• micro-batch size 1
• gradient accumulation 8
• effective batch size 8
• cosine scheduler
• max sequence length 512

Method Implementations

1. LoRA FP16

LoRA is used as the main full-precision PEFT baseline.

The original model weights are frozen, and low-rank adapter matrices are added to selected transformer layers.

The LoRA update can be represented as:

[ W_{LoRA} = W_0 + BA ]

Where:

• (W_0) is the frozen pretrained weight
• (BA) is the trainable low-rank update

LoRA is applied in FP16 precision.

2. QLoRA 4-bit

QLoRA is used as the main quantized PEFT baseline.

The base model is loaded in 4-bit precision, while the LoRA adapters remain trainable.

This significantly lowers GPU memory requirements compared to FP16 LoRA.

3. AdaLoRA 4-bit

AdaLoRA is used to test adaptive rank allocation under 4-bit quantization.

The model is first loaded in 4-bit precision, then prepared for k-bit training. AdaLoRA adapters are added, and rank allocation is updated dynamically during training.

Pseudo-code

from peft import AdaLoraConfig, get_peft_model, prepare_model_for_kbit_training

model = load_model_4bit(BASE_MODEL)
model = prepare_model_for_kbit_training(model)

adalora_config = AdaLoraConfig(
    task_type="CAUSAL_LM",
    r=16,
    target_r=8,
    deltaT=10
)

model = get_peft_model(model, adalora_config)

for batch in train_loader:
    optimizer.zero_grad()
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    model.base_model.update_and_allocate(step)
    optimizer.step()

evaluate(model)

4. DyLoRA 4-bit

DyLoRA is used as a dynamic-rank baseline.

Since DyLoRA does not have a standard implementation in the PEFT library, a DyLoRA-style approximation is used.

Pseudo-code

# Load quantized base model
model = load_base_model(load_in_4bit=True)
model = prepare_model_for_kbit_training(model)

# Add LoRA adapters with maximum rank
config = LoraConfig(r=r_max, ...)
model = get_peft_model(model, config)

# Training loop
for step, batch in train_loader:

    # sample active rank
    r_active = random_integer(1, r_max)

    # apply dynamic rank masking
    for each LoRA layer:
        use only first r_active columns of A and rows of B

    # forward + backward
    loss = model(batch)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

# evaluation
evaluate(model)

The adapter is trained so that different rank budgets can be tested without training separate models for each rank.

Fairness in Comparison

To ensure a fair comparison between PEFT methods:

• The same base model is used
• The same dataset split is used
• The same preprocessing pipeline is used
• The same training configuration is used
• The same evaluation procedure is used

For quantized methods:

• The base model is loaded in 4-bit precision

For LoRA:

• The base model remains in 16-bit precision

Performance is evaluated using:

• Validation perplexity
• WikiText-2 perplexity
• GPU memory usage
• Training runtime

Evaluation Metrics

The main metric used is perplexity (PPL).

Lower perplexity values indicate better performance.

Perplexity is computed as:

[ PPL = e^{loss} ]

Two types of perplexity are used:

• Alpaca validation perplexity
• WikiText-2 perplexity

Additional metrics include:

• GPU memory usage
• Memory reduction percentage
• Training runtime
• Number of trainable parameters
• Efficiency score

The efficiency score is computed as:

[ Efficiency\ Score = \frac{1} {PPL \times GPU\ Memory \times Training\ Runtime} ]

All experiments are benchmarked using a unified Google Colab notebook to ensure consistent evaluation conditions.

Results

The comparison focuses on:

• Model quality
• GPU memory usage
• Training runtime
• Overall efficiency

Main Observations

• LoRA achieves the best overall perplexity on both benchmarks.
• QLoRA and the DyLoRA-style implementation remain close to LoRA with only small degradation.
• AdaLoRA achieves the weakest perplexity results.

WikiText-2 Perplexity

• LoRA: 7.62
• AdaLoRA: 8.59

Alpaca Validation Perplexity

• LoRA: 3.2
• AdaLoRA: 4.62

GPU Memory Usage

QLoRA achieves the lowest GPU memory usage.

The DyLoRA-style implementation also maintains relatively low memory consumption.

AdaLoRA consumes more memory than expected due to adaptive rank allocation overhead.

Efficiency Score

QLoRA achieves the highest efficiency score.

The DyLoRA-style method achieves the second-best efficiency score.

LoRA maintains the best model quality but suffers from higher memory usage.

AdaLoRA achieves efficiency similar to LoRA despite weaker perplexity performance.

Runtime Analysis

Training runtime comparisn:

• LoRA FP16: ~387 seconds
• QLoRA: ~630 seconds
• DyLoRA-style: ~698 seconds
• AdaLoRA: ~954 seconds

One important observation is that lower GPU memory usage does not necessarily imply shorter training time.

Quantization introduces:

• quantization overhead
• dequantization overhead
• additional computations

Therefore, QLoRA primarily improves memory efficiency rather than training speed.

Discussion

The results show that no single PEFT method performs best across all metrics simultaneously.

LoRA achieves the strongest perplexity results because it operates on full-precision weights.

QLoRA provides the strongest balance between:

• performance
• memory efficiency

The DyLoRA-style implementation achieves competitive performance while maintaining reduced memory usage.

AdaLoRA shows weaker performance under the current setup. One possible explanation is that adaptive rank allocation requires:

• more training data
• more training steps
• better convergence conditions

Since the experiments are constrained by Google Colab computational limitations, the adaptive rank redistribution process may not stabilize properly.

Limitations

The study is constrained by several limitations:

• The experiments are performed on a relatively small model
• Training steps are limited
• Only a single seed is used
• The DyLoRA-style implementation is not an official reproduction

Future work may extend the experiments to:

• larger LLMs
• larger datasets
• more advanced hardware
• more complete dynamic-rank implementations

Conclusion

This paper compares several PEFT methods including:

• LoRA
• QLoRA
• AdaLoRA
• DyLoRA-style implementation

The experiments evaluate the methods using:

• downstream-task perplexity
• general language perplexity
• GPU memory usage
• training runtime

The results show that:

• LoRA achieves the best perplexity
• QLoRA provides the best performance-memory trade-off
• DyLoRA-style implementation remains competitive
• AdaLoRA performs weaker under the current setup

The experiments show that quantized PEFT methods can significantly reduce GPU requirements while maintaining relatively competitive performance under practical limited-scale experimental conditions.

References

1. Hu et al. (2022). LoRA: Low-rank adaptation of large language models. ICLR.
2. Dettmers et al. (2023). QLoRA: Efficient finetuning of quantized LLMs. NeurIPS.
3. Zhang et al. (2023). AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. ICLR.
4. Valipour et al. (2023). DyLoRA: Parameter-efficient tuning using dynamic search-free low rank adaptation. EACL.
5. Hugging Face. (2024). PEFT Library.
6. Taori et al. (2023). Stanford Alpaca.
7. Touvron et al. (2023). LLaMA: Open and efficient foundation language models.
8. Houlsby et al. (2019). Parameter-efficient transfer learning for NLP. ICML.
9. Zhang et al. (2024). TinyLlama: An Open-Source Small Language Model.