LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models

Smaller Adapters can achieve better performance

NAACL 2024, Oral

21 Apr 2024

[paper] [code]

Various parameter-efficient fine-tuning (PEFT) techniques have been proposed to enable computationally efficient fine-tuning while maintaining model performance. However, existing PEFT methods are still limited by the growing number of trainable parameters with the rapid deployment of Large Language Models (LLMs). To address this challenge, we present LoRETTA, an ultra-parameter-efficient framework that significantly reduces trainable parameters through tensor-train decomposition. Specifically, we propose two methods, named LoRETTA_adp and LoRETTA_rep. The former employs tensorized adapters, offering a high-performance yet lightweight approach for the fine-tuning of LLMs. The latter emphasizes fine-tuning via weight reparameterization with a set of small tensor factors.

The performance vs. trainable parameters on the DeBERTa-Base model, showcasing the strong performance of LoRETTA methods with ultra parameter efficiency across various GLUE tasks (Average scores).

Based on the structure of Adapters and LoRA methods, we employ the idea of tensor-train decomposition to formulate the linear layers into tensorized layers and show smaller Adapters can even achieve better performance by using such factorization methods with larger compression ratios. LoRETTA achieves comparable or better performance than most widely used PEFT methods with up to 100× fewer parameters on the LLaMA-2–7B models. Furthermore, empirical results demonstrate that the proposed methods exhibit remarkable anti-overfitting capability, effectively improve training efficiency, and enjoy better multi-task learning performance. Plug-and-play loretta library built upon the Huggingface framework and PEFT library are provided.

Performance Comparison on LLaMA-2–13B and LLaMA-2–70B. We compare our proposed method with LoRA and IA3, which are the most widely used high-performance PEFT methods. Comparisons are made among various Natural language understandings (NLU) and Questions Answering (QA) tasks.

Motivation

Even though Adapters (Houlsby et al., 2019) and LoRA (Hu et al., 2021) methods have been widely used for fine-tuning large language models (LLMs), there are still three concerns that have not been fully addressed in previous works.

First, these two types of methods still incur a large number of trainable parameters as model sizes increase. Taking the LLaMA-2–70B model as an example, LoRA needs to update over 16 million parameters, and Adapters methods require over 250 million parameters. This number is even greater than the total parameters in some BERT models. Thus, we are considering further reductions in the number of trainable parameters by employing factorization methods with larger compression ratios. (refer to the figure above)

Second, the LoRA and Adapters methods have a serious overfitting issue. With more trainable parameters, both methods suffer from increased overfitting, which negatively impacts their performance. We observe that by reducing the number of trainable parameters, our proposed methods demonstrate strong anti-overfitting capabilities, which is a key factor in improving performance with fewer trainable parameters.

Evaluation loss comparison across various PEFT methods on the DeBERTa-base model. The loss is smoothed with a window size of 20 and the shallow means the standard deviation boundaries.

Third, according to recent experimental observations (Hu et al., 2023), the full-size series Adapters method actually outperforms the LoRA method across a wide variety of tasks. This superiority is attributed to the non-linearity in the injected adapters used by the Adapters method. The main drawback of the Adapters method, when compared to LoRA, is the significant number of trainable parameters due to the numerous adapter layers. Our method presents a viable alternative to the traditional Adapters approach by utilizing tensorized layers to substantially reduce the number of trainable parameters.

Accuracy comparison of LLMs with different adapters on eight commonsense reasoning datasets. We can observe the series adapters and parallel adapters method perform better than LoRA method on GPT-J and Llama-13B models. I guess the fine-tuning of layer norm and the added non-linearity layer helps a lot.

LoRETTA Methods

PEFT methods can be broadly categorized into three types, the adapters, the reparameterization method, and the prompt-based method (Hu et al., 2023). Among them, the reparameterization-based (like LoRA) and adapter-based methods (like Adapters) are notable for incorporating new structures within the model architecture, thereby introducing a large number of additional trainable parameters. To reduce the size of the injected modules, we introduce our LoRETTA framework, which contains the adapter-based approach LoRETTA_adp and the reparameterization based approach LoRETTA_rep .

LoRETTA_rep

: Architecture of the LoRETTArep method for a single transformer encoder.

The idea of the reparameterization also appeared in LoRA, which updates the weight with two low-rank matrices in a linear layer as follows:

where x and y denote the input and output of a linear layer. Setting h as the hidden size of the model, W_0 is a pre-trained weight matrix.

Here, we propose a more compact way to represent the updating matrix with two tensorized layers (without bias terms), whose general idea is depicted in the above figure. In our method, we also employ the bottleneck structure to first reduce the large updating matrix into two small matrices. Then, we reshape the two updating matrices ∆W_up and ∆W_down into tensors with the shape of k1 × · · · × kd and j1 × · · · × jd. Here, both tensors are cast into TT factors (a sequence of tensors). During the training process, we only update and store the small tensor factors instead of original weight matrices. The tensorized update process of a full-connected layer with linear transformation to an input x can be expressed as:

In this manner, our approach reduces the parameters from 12K to 1K for a single reparameterization adapter compared with the LoRA method with the LoRA rank of 8, when the hidden size is 768 and the tensor rank is 5 for the LoRETTA_rep method.

LoRETTA_adp

Similar as LoRETTA_rep, our tensorized adapters achieve a much larger compression ratio by including two tensorized linear layers and an activation function. For example, set the hidden size of the models as 768, and the bottleneck size as 64, compared to the Adapters method with the number of trainable parameters of 2 · 768 · 64 ≈ 98K for weight matrices, LoRETTA_adp adds only 1.2K parameters, assuming tensor shapes of [8, 8, 8, 8, 8, 8] and a constant TT rank of 5.

The tensorized method aligns well with the Adapters structure, significantly reducing the number of trainable parameters by compressing the weight matrix with a high compression ratio. This reduction greatly enhances the anti-overfitting capabilities of these methods. At the same time, the added non-linearity is retained, preserving the advantages of traditional adapters compared to the LoRA method.

The architecture of LoRETTA_adp for the transformer encoders or decoders. the tensorized classifier is optional for different tasks. For classification tasks, we set this part to be trainable and we freeze this part during language modeling tasks.*

When to use LoRETTA?

Based on our observations, the LoRETTA_adp method can save about four times the number of trainable parameters compared to the LoRA method, while the LoRETTA_rep method can save up to eight times. Performance-wise, the LoRETTA_adp and LoRETTA_rep methods perform similarly on Bert-base models. Therefore, we recommend using the LoRETTA_rep method for medium-sized models like BERT. In tests on LLaMA-2–7B/13B/70B models, LoRETTA_adp achieves better performance than all other parameter-efficient fine-tuning (PEFT) methods. Consequently, we highly recommend using LoRETTA_adp for larger models.

Generally, LoRETTA_adp performs better, but its effectiveness depends on the specific models, tasks, and hardware constraints, especially if you require a PEFT method with exceptionally low trainable parameters. We strongly encourage you to try our LoRETTA package. Simply install the package and add a few lines of code to integrate it with the Huggingface model using methods similar to the official PEFT package.

How to use LoRETTA?

We provide a detailed introduction and straightforward sample code for testing our LoRETTA methods in our GitHub repository. Feel free to explore and give it a try. If you encounter any issues while experimenting with the code, please do not hesitate to report them to us. We are eager to assist and resolve any problems you may have!