Complete Guide to LLM Finetuning: SFT, RLHF, DPO, ORPO with PEFT and LoRA

Theory, mathematical intuition, and practical implementation — everything you need to go from base model to production-ready LLM.

Introduction — Why Finetuning is Needed

The modern AI landscape is defined by large, general-purpose language models — GPT-4, LLaMA 3, Mistral, Gemma — that have been pretrained on trillions of tokens of internet text. These models are extraordinarily capable: they can write code, summarize documents, answer complex questions, and even reason through multi-step problems. Yet, for the vast majority of real-world production applications, base pretrained models simply are not enough.

This guide walks through every major finetuning technique used in production today: Supervised Fine-Tuning (SFT), RLHF, DPO, and ORPO — and how Parameter-Efficient Finetuning (PEFT) with LoRA makes all of it accessible on consumer hardware.

All Code Available — Train Your Own Model Today

If you want to skip straight to the code and start training your own model — no need to read the full article. Just head to the GitHub repo, pick your method, and run the notebook.

Every technique covered in this guide has a fully working, tested implementation in my GitHub repository. All notebooks which use PEFT and LORA were trained and verified on Google Colab — no expensive cloud setup required.

👉 **github.com/anilnishad19799/peft-lora-llm-finetuning**

I have trained all PEFT + LoRA models on Google Colab — every notebook runs out of the box. Clone the repo, pick your method, and start training.

What are Foundation Models / Base LLMs?

A base LLM is a neural network (typically a transformer decoder) that has been pretrained on a massive, diverse text corpus using next-token prediction. During pretraining, the model learns language structure, world knowledge, reasoning patterns, and stylistic conventions — all encoded into billions of parameters.

Think of a base LLM as a brilliant but undisciplined university graduate who has read almost every book ever written, but has never had a real job. They know a lot — but they don’t know how to behave in a specific professional context.

🎓 Base models are powerful general learners, but they are NOT ready-to-deploy products.

Limitations of Base Models

• Generic responses: A base model predicts the statistically likely next token, not the most helpful answer. Ask it a question and it may respond with another question, or simply continue the prompt as if writing a book.
• Lack of domain knowledge: A base model trained on general internet data has shallow knowledge of your specific domain — medical imaging, legal contracts, financial filings, or internal company documentation.
• Alignment issues: The model has no intrinsic motivation to be helpful, harmless, or honest. It might generate harmful content, incorrect facts, or evasive non-answers.
• Hallucinations: Without grounding mechanisms, base LLMs confidently generate plausible-sounding but factually incorrect information — a critical problem in high-stakes domains like healthcare or law.

Pretraining vs. Finetuning vs. Alignment

Pretraining is where the model is trained on trillions of tokens using next-token prediction on raw internet text. The goal is to learn general language structure and world knowledge.

Finetuning continues training on a smaller, task-specific dataset. The goal is to adapt the model’s behavior for a specific task or domain — like medical Q&A or code generation.

Alignment optimizes the model to follow human preferences and instructions. The goal is to make the model helpful, harmless, and honest — this is where SFT, RLHF, DPO, and ORPO come in.

The Two Training Paradigms: Full Fine-Tuning vs PEFT + LoRA

A. Full Model Finetuning

Full finetuning updates every single parameter in the pretrained model. You take the base model’s weights and continue gradient descent on your task-specific dataset — modifying all layers from the embedding layer to the final LM head.

Your optimizer (AdamW) computes gradients for every parameter. The learning rate is typically much lower than pretraining (1e-5 to 5e-5) to avoid catastrophic forgetting — destroying the general knowledge learned during pretraining.

# Full finetuning — every parameter is updated
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")
optimizer = AdamW(model.parameters(), lr=2e-5)

for batch in dataloader:
    loss = model(**batch).loss
    loss.backward()
    optimizer.step()  # updates ALL ~8 billion parameters

When To Use Full Finetuning

• Large organizations: That have A100/H100 clusters and want maximum performance on a specific domain.
• Critical production use cases: Medical, legal, or financial NLP where domain accuracy is paramount.
• When you have very large domain datasets: Millions of tokens of high-quality data.

VRAM requirements: A 7B model requires ~14 GB for weights in fp16, ~56 GB for optimizer states (AdamW), ~14 GB for gradients — totaling 80+ GB, which means 2–4× A100s minimum.

B. Parameter-Efficient Finetuning (PEFT)

Full finetuning is brutally expensive. PEFT solves this by freezing the original model weights and only training a small number of additional parameters — often less than 1% of the total model size.

LoRA: Low-Rank Adaptation

LoRA (Hu et al., 2021) is the most widely adopted PEFT method. The core insight: changes in model weights during adaptation tend to lie in a low-dimensional subspace. Instead of updating the full weight matrix W ∈ ℝd×k, LoRA decomposes the update into two small matrices:

ΔW = B × A where B ∈ ℝ^(d×r), A ∈ ℝ^(r×k), and r << min(d, k)

During training, only A and B are updated. The original weight W stays frozen. During the forward pass:

h = Wx + (B × A)x × α/r (α is a scaling factor)

from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                    # rank — lower = less memory
    lora_alpha=32,           # scaling factor
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 8,388,608 || all params: 6,746,804,224 || trainable%: 0.12

Memory Savings with LoRA

• A 7B parameter LLaMA model: Full finetuning requires ~112 GB (weights + gradients + optimizer states). With LoRA (rank=16), trainable parameters drop to ~8M — requiring less than 16 GB.

• Rank selection: r=4 to r=64 are common choices. Higher rank = more expressivity, more memory.

• Target modules: LoRA is typically applied to attention projection matrices (q_proj, v_proj, k_proj, o_proj) and sometimes MLP layers.

Fine-Tuning Methods

SFT — Supervised Fine-Tuning

SFT is the simplest and most foundational fine-tuning method. You provide the model with (instruction, response) pairs and train it to minimize the cross-entropy loss on the response tokens:

L_SFT = -Σ log P(yi | x, y<i), where y = response tokens, x = prompt

The model learns to complete instructions the way your dataset demonstrates. This is how ChatGPT was initially trained — on human-written demonstrations of helpful responses.

Dataset format (instruction tuning):

{
  "instruction": "Summarize the following text in 2 sentences.",
  "input": "The Eiffel Tower was built in 1889...",
  "output": "The Eiffel Tower, completed in 1889, is an iron lattice tower in Paris..."
}

RAM Consumption:

| Model Size | Full SFT (fp32) | Full SFT (bf16) | SFT + LoRA (4-bit) |
|------------|-----------------|-----------------|-------------------|
| 1B         | ~16 GB          | ~8 GB           | ~3 GB             |
| 7B         | ~112 GB         | ~56 GB          | ~6–8 GB           |
| 13B        | ~208 GB         | ~104 GB         | ~10–14 GB         |
| 70B        | >1 TB           | ~500 GB         | ~48 GB            |

Limitations of SFT:

• Quality depends entirely on dataset quality
• Model can mimic bad demonstrations
• No mechanism to prefer one good response over another
• Doesn’t explicitly penalize harmful outputs

Implementation Links:

• 🔗 Full Model SFT: https://github.com/anilnishad19799/peft-lora-llm-finetuning/sft_with_trl.ipynb
• 🔗 SFT with PEFT + LoRA: https://github.com/anilnishad19799/peft-lora-llm-finetuning/sft_with_peft_lora.ipynb

RLHF — Reinforcement Learning from Human Feedback

RLHF is the technique behind ChatGPT and Claude. Rather than imitating demonstrations, RLHF optimizes the model to maximize human preference scores — captured by a learned reward model.

The RLHF Pipeline: 3 Stages

1. SFT Model Training : Finetune the base model on high-quality instruction-following data. This creates the SFT model — the starting point for RLHF.
2. Reward Model Training : Human annotators rank pairs of model responses. A reward model R(x, y) is trained to predict preference scores using the Bradley-Terry model.

L_RM = -log(σ(r(x, y_w) — r(x, y_l)))

Where y_w is the preferred (winner) response and y_l is the dispreferred (loser) response.

3. PPO Optimization: Use Proximal Policy Optimization (PPO) to fine-tune the SFT model to maximize the reward model’s score, while staying close to the original SFT model (KL penalty to prevent reward hacking):

L_PPO = E[r(x,y)] — β * KL(π_θ || π_ref)

RAM Consumption:

RLHF is notoriously memory-hungry because you need to hold 4 models in memory simultaneously:

1. Policy model (being trained)
2. Reference model (frozen SFT model for KL)
3. Reward model
4. Value/critic model

| Model Size | RLHF Full   | RLHF + LoRA  |
|------------|-------------|--------------|
| 7B         | ~320+ GB    | ~24–40 GB    |
| 13B        | ~600+ GB    | ~40–60 GB    |

This is why RLHF is often impractical without a large GPU cluster. Libraries like TRL by Hugging Face implement memory-efficient PPO, and DeepSpeed ZeRO helps distribute the load.

Limitations of RLHF:

• Extremely complex pipeline (3 training stages)
• Reward model can be gamed (reward hacking)
• Unstable training (PPO is notoriously finicky)
• High memory and compute cost
• Requires large amounts of human preference labels

Implementation Links:

• 🔗 Full SFT + RLHF: https://github.com/anilnishad19799/peft-lora-llm-finetuning/sft_with_rlhf_PPO.ipynb

DPO — Direct Preference Optimization

DPO (Rafailov et al., 2023) is one of the most important recent advances in LLM alignment. The key insight: the RLHF objective can be rearranged analytically to express the optimal policy directly in terms of preference data — eliminating the reward model and PPO entirely.

The key insight is that the optimal policy under RLHF has a closed-form solution. By reparameterizing, DPO derives a loss function that trains directly on (prompt, chosen, rejected) triplets — no separate reward model, no PPO:

L_DPO = -E[ log σ( β log(π_θ(y_w|x)/π_ref(y_w|x)) — β log(π_θ(y_l|x)/π_ref(y_l|x)) ) ]

Intuition: The model is pushed to assign higher probability to the chosen response relative to the reference model, and lower probability to the rejected response.

{
  "prompt": "What is the capital of France?",
  "chosen": "The capital of France is Paris.",
  "rejected": "France doesn't have a capital."
}

RAM Consumption:

DPO only needs 2 models (policy + frozen reference), making it dramatically cheaper than RLHF:

| Model Size | DPO Full  | DPO + LoRA |
|------------|-----------|------------|
| 7B         | ~160 GB   | ~14–20 GB  |
| 13B        | ~300 GB   | ~24–36 GB  |

Advantages over RLHF:

• Single training stage (no reward model needed)
• Much more stable training
• Mathematically equivalent to RLHF under certain assumptions
• Widely used in practice (Llama-2-Chat, Zephyr, etc.)

Limitations:

• Still requires a reference model (memory overhead vs SFT)
• Sensitive to the quality of preference pairs
• Can degrade on tasks not covered by preferences

Implementation Links:

• 🔗 Full Model DPO: https://github.com/anilnishad19799/peft-lora-llm-finetuning/dpo_with_trl.ipynb
• 🔗 DPO with PEFT + LoRA: https://github.com/anilnishad19799/peft-lora-llm-finetuning/dpo_with_peft_lora_trl.ipynb

ORPO — Odds Ratio Preference Optimization

ORPO (Hong et al., 2024) takes efficiency one step further. DPO still requires a separate SFT phase followed by preference tuning. ORPO asks: Can we do SFT and preference alignment in a single step?

ORPO introduces a novel odds ratio term added directly to the SFT loss:

L_ORPO = L_SFT + λ * L_OR

Where the odds ratio loss is:

L_OR = -log σ( log( OR_θ(y_w|x) / OR_θ(y_l|x) ) )

The odds ratio OR_θ(y|x) = P(y|x) / (1 − P(y|x)) captures how much more likely the model is to generate a response than not. By penalizing the model when the odds of a rejected response approach those of a preferred one, ORPO simultaneously teaches instruction following and preference alignment — in one pass.

What this does:

• The SFT term trains the model to generate chosen responses well
• The OR term penalizes the model for generating rejected responses
• No reference model needed at all — only one model in memory

RAM Consumption:

ORPO is the most memory-efficient alignment method:

| Model Size | ORPO Full    | ORPO + LoRA |
|------------|--------------|-------------|
| 7B         | ~80–100 GB   | ~8–12 GB    |
| 13B        | ~160 GB      | ~14–20 GB   |

Advantages:

• Single training pass for SFT + alignment
• No reference model (saves significant memory vs DPO)
• Simpler pipeline
• Competitive performance with DPO and RLHF

Limitations:

• Newer method — less community validation than DPO/RLHF
• Requires both positive (chosen) and negative (rejected) examples for all training data
• The λ hyperparameter requires tuning

Implementation Links:

• 🔗 Full Model ORPO: https://github.com/anilnishad19799/peft-lora-llm-finetuning/orpo_with_trl.ipynb
• 🔗 ORPO with PEFT + LoRA: https://github.com/anilnishad19799/peft-lora-llm-finetuning/orpo_with_peft_lora_trl.ipynb

Part 3 — Comprehensive Comparison

3.1 Method Complexity & Pipeline

3.2 Memory Comparison (7B Model)

3.3 Training Stability

3.4 Output Quality & Alignment

3.5 Use Case Recommendations

GPU Requirements & Full Method Comparison

Conclusion

Fine-tuning LLMs has evolved dramatically — from expensive full-parameter updates to elegant single-stage alignment:

• SFT is your starting point — always required, efficient with LoRA
• RLHF gives the best alignment but at enormous cost
• DPO democratized preference learning — the current industry standard
• ORPO is the new challenger — one pass, no reference model, surprisingly competitive

The combination of PEFT + LoRA (or QLoRA) has been the true game-changer, bringing 7B–13B model fine-tuning to a single consumer GPU. There’s no reason today to train full parameters unless you’re at a very large scale.

The field is moving fast — methods like IPO, KTO, and SPIN are worth watching. But for most practitioners, mastering SFT + DPO/ORPO with LoRA is more than enough to build world-class models.

Resources & References

• LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021
• QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023
• InstructGPT / RLHF — Ouyang et al., 2022
• Direct Preference Optimization — Rafailov et al., 2023
• ORPO: Monolithic Preference Optimization — Hong et al., 2024
• Hugging Face TRL Library
• PEFT Library

If you found this useful, please clap and follow — I post deep dives on LLMs, training infrastructure, and practical ML every week.