Supercharging LLM Fine-Tuning: Unsloth, LoRA, and ORPO for Travel Q&A Classification on a Budget

Fine-tuning Large Language Models (LLMs) unlocks incredible potential, but it’s often perceived as computationally expensive and complex. How can we make it faster, more memory-efficient, and achieve state-of-the-art results, especially on resource-constrained platforms like Google Colab or Kaggle?

This post dives into a practical project where we tackled classifying travel-related questions using modern, efficient techniques. We’ll explore how Unsloth dramatically speeds up training, how LoRA enables efficient parameter tuning, and how ORPO (Odds Ratio Preference Optimization) simplifies the fine-tuning process while significantly boosting performance.

Follow along as we transform capable baseline models into highly accurate travel question classifiers, demonstrating that powerful fine-tuning is within reach even without high-end hardware.

Complete in-depth code and comparison can be found in my Github: https://github.com/LupraV/TravelQA_Classification_LLM-SFT

The Challenge: Classifying Travel Questions

Imagine needing to automatically categorize user questions on a travel forum or chatbot. Are they asking about “Things to Do” (TTD), “Accommodation” (ACM), “Transport” (TRS), or something else?

Dataset: 5000 travel domain based questions developed for the research paper “Question Answering system for the travel domain”:

Kahaduwa, H. et al., 2017. Question Answering system for the travel domain. In Engineering Research Conference (MERCon), 2017 Moratuwa (pp. 449–454). IEEE.

Our starting point are 5000 travel questions, each labeled with one of seven categories:

TTD: Things to Do

TGU: Travel Guide

ACM: Accommodation

TRS: Transport

WTH: Weather

FOD: Food

ENT: Entertainment

The dataset has a somewhat imbalanced distribution, as shown below, making the task slightly more challenging.

Our goal is to train an LLM to accurately predict the correct category for any given travel question, focusing on models that balance performance and size for feasibility on free-tier platforms.

The Toolkit: Efficiency is Key

To tackle this efficiently, we leveraged a powerful combination of tools and models:

The Models: Phi-3.5-mini & Gemma-2–9B (mainly, also tested on others)

• Why these? We selected Microsoft’s Phi-3.5-mini-instruct (3.8B) and Google’s Gemma-2–9B-instruct (9B) based on their impressive balance of performance (strong rankings on leaderboards like Open LLM Leaderboard), size, and compatibility with Unsloth for running on free Colab/Kaggle GPUs.
• Phi-3.5-mini: Despite its small size, it boasts a massive 128K token context length (pre-trained on 3.3T tokens) and strong reasoning capabilities. It uses a LlamaTokenizer (32k vocab) and likely employs optimizations like GeGLU activation and Grouped-Query Attention (GQA). It was fine-tuned using SFT and DPO.
• Gemma-2–9B: Offers stronger reasoning for complex tasks with its larger size but has a shorter 8K context length. It uses RoPE, alternates between local sliding window and global attention, has a large 256k vocabulary (pre-trained on 6T tokens), and was fine-tuned with SFT and RLHF.

Unsloth: This library was crucial. It enabled us to run these models effectively by:

• Speeding up training: Delivering significantly faster fine-tuning (matching their 2x claim in our experience).
• Reducing memory: Primarily through seamless integration with bitsandbytes for 4-bit quantization (load_in_4bit=True). This allowed even the 9B Gemma model to run (mostly) within the 15GB vRAM limits of Colab’s T4 GPU, albeit with a minor (1–5%) performance dip compared to non-quantized versions in pre-training tests.
• Optimizing Precision: Using dtype=None and fp16=not is_bfloat16_supported() allowed automatic selection of fp16 precision on T4/P100 GPUs, further optimizing speed and memory.

LoRA (Low-Rank Adaptation): A Parameter-Efficient Fine-Tuning (PEFT) technique. We used LoRA (via Unsloth’s get_peft_model) to introduce small, trainable matrices, drastically reducing the number of parameters needing updates. We chose a rank (r) of 16 as a balance between performance potential and memory efficiency, crucial for the constrained environment.

ORPO (Odds Ratio Preference Optimization): This technique simplifies the alignment process by combining supervised fine-tuning (SFT) and preference alignment (like DPO) into a single step using an odds ratio-based loss function. This avoids separate SFT and DPO stages, making the workflow more efficient.

The Process: From Baseline to Fine-Tuned Model

Here’s a breakdown of our approach:

1. Setup and Data Preparation:

• Installed necessary libraries: unsloth, trl, peft, transformers, datasets, bitsandbytes, etc.
• Loaded, cleaned, and split the travel Q&A dataset into stratified training (4000), validation (300), and test (700) sets.

2. Model Selection and Loading (with Unsloth):

• Loaded the chosen models (unsloth/Phi-3.5-mini-instruct, unsloth/gemma-2–9b-it-bnb-4bit) using FastLanguageModel, enabling 4-bit quantization and auto-detecting the optimal dtype (fp16 on T4).

from unsloth import FastLanguageModel
import torch

model_name = "unsloth/Phi-3.5-mini-instruct" # Or Gemma
max_seq_length = 512
dtype = None # Auto-detect
load_in_4bit = True # USE 4-bit quantization

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = model_name,
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

3. Baseline Evaluation (Zero-Shot & Few-Shot):

Prompt Engineering: We iterated through several prompt designs:

• Prompt 1 (Basic): Minimal input, performed poorly.
• Prompt 2 (Concise): Added class descriptions, improved results but still limited.
• Prompt 3 (Intent-focused): Refined to capture intent rather than relying on keywords. Performed well, especially for Phi-3.5-mini (reaching 85–90% on sample tests).
• Prompt 4 (CoT): Introduced step-by-step Chain-of-Thought reasoning, significantly boosting performance for the larger Gemma-2–9B model (90–100% on sample tests). This highlighted the need to match prompt complexity with model capacity.

# STEP 4: PROMPT Engineering
# https://www.promptingguide.ai/techniques
# https://arxiv.org/abs/2305.08291
# https://arxiv.org/abs/2302.12822
# https://arxiv.org/abs/2402.07927
# https://dottxt-ai.github.io/outlines/ Did not use but could potentially improve results.

# Formatting function for prompts (for training)
def formatting_prompts_func_training(example):
    text = f"### Question: {example['question']}\n### Answer: {example['class']}"
    return {'text': text}

# Formatting function for prompts (for evaluation)
def formatting_prompts_func_eval(example):
    text = f"### Question: {example['question']}"
    return {'text': text}

# Apply formatting to datasets
train_dataset = train_dataset.map(formatting_prompts_func_training)
val_dataset = val_dataset.map(formatting_prompts_func_eval)
test_dataset = test_dataset.map(formatting_prompts_func_eval)

Testing Setup: Used deterministic settings suitable for classification: temperature=0.1, top_p=0.9, top_k=5, and max_new_tokens=5 (to ensure only the 3-letter class code was generated). Initial tests with do_sample=False yielded identical results, confirming the model’s deterministic nature at low temperatures.

Baseline Performance: Evaluated on the full test set using the best-performing prompts before fine-tuning.

• Zero-shot accuracy varied (see table below). Few-shot (especially 1-shot) results were inconsistent, sometimes hurting performance if the example was ambiguous.

Initial Observations: Models often relied heavily on keywords, leading to misclassifications when context was nuanced (e.g., confusing Food/FOD and Entertainment/ENT based on “restaurants” or “bars”). Overlapping classes (like TTD/TGU) were challenging. No major hallucinations were observed.

4. Fine-Tuning with Unsloth, LoRA, and ORPO:

• LoRA Configuration: Applied LoRA with r=16 to the attention and MLP modules using FastLanguageModel.get_peft_model.

# LoRA Adapters with PEFT
model = FastLanguageModel.get_peft_model(
    model,
    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

• ORPO Data Formatting: Created a function to structure the data with prompt, chosen (correct class + EOS token), and rejected (concatenated incorrect classes + EOS tokens) fields, using the best-performing prompt template (Prompt 3 for initial Phi-3.5, Prompt 4 Old/New for later tests).

# Custom ORPO classification prompt based on the best prompt template
#PROMPT #3
orpo_prompt = f"""Given the following travel related question and descriptions of possible classes {descriptions}.
Use reasoning to identify the question's overall intent and type of information being asked, to determine which class among {possible_classes} the question best fits into.
Answer MUST contain ONLY the class 3-letters, nothing else.

Question: {{}}
###Answer:"""

EOS_TOKEN = tokenizer.eos_token

# Formatting the training, validation, and test sets for ORPO
def format_orpo_prompt(sample):
    question = sample["question"]
    correct_class = sample["class"]
    incorrect_classes = [cls for cls in class_descriptions.keys() if cls != correct_class]
    # ORPOTrainer expects 'prompt', 'chosen', and 'rejected' keys
    sample["prompt"] = orpo_prompt.format(question=question) # Corrected format call
    sample["chosen"] = correct_class + tokenizer.eos_token # Correct class with EOS token
    sample["rejected"] = ' '.join([cls + tokenizer.eos_token for cls in incorrect_classes])
    return sample

#Apply the formatting function to all datasets
train_dataset_orpo = Dataset.from_pandas(train_data).map(format_orpo_prompt)
val_dataset_orpo = Dataset.from_pandas(val_data).map(format_orpo_prompt)
test_dataset_orpo = Dataset.from_pandas(test_data).map(format_orpo_prompt)

• ORPO Trainer Setup: Used trl.ORPOTrainer with ORPOConfig. Key hyperparameters included beta=0.1, learning_rate=1e-4 or 2e-4, optim=”adamw_8bit”, and max_steps=100 (or 200 for one Gemma test). Unsloth automatically patched the trainer.

# ORPO - Configuration and Trainer Setup
PatchDPOTrainer() # Patch for DPO Trainer
# ORPO Trainer Setup
orpo_trainer = ORPOTrainer(
model=model,
train_dataset=train_dataset_orpo, # Training dataset with chosen/rejected
eval_dataset=val_dataset_orpo, # Validation dataset for evaluation during␣
↪fine-tuning
tokenizer=tokenizer,
args=ORPOConfig(
beta=0.1,
48
max_length=1024,
max_prompt_length=512,
max_completion_length=56,
per_device_train_batch_size=4,
per_device_eval_batch_size=4,
gradient_accumulation_steps=2,
learning_rate=1e-4, #2e-4
logging_steps=1,
optim="adamw_8bit"
,
weight_decay=0.01,
lr_scheduler_type="linear"
, # "cosine"
#num_train_epochs = 1, # 1 for full training run
max_steps=100, # None for full run (comment out if using␣
↪num_train_epochs)
fp16=not is_bfloat16_supported(), # Use FP16 if BFloat16 isn't␣
↪supported
bf16=is_bfloat16_supported(), # Use BFloat16 if available
output_dir="./orpo_results"
,
report_to="none"
, # "wandb"
),
)
# Begin ORPO fine-tuning
orpo_trainer.train()

The Results: Dramatic & Efficient Accuracy Gains

The ORPO fine-tuning process yielded significant improvements rapidly. Here’s a summary comparing pre-training (zero-shot using best prompts) and post-ORPO fine-tuning results on the test set:

Key Observations:

• Massive Improvement: ORPO fine-tuning delivered substantial accuracy gains (+17~24 absolute percentage points) across models and prompts.
• Efficiency: These gains were achieved in just 100–200 training steps, demonstrating the power and speed of the Unsloth + LoRA + ORPO stack on limited hardware.
• Model Capacity Matters: The larger Gemma-2–9B consistently outperformed Phi-3.5-mini, achieving a peak accuracy of 87.71%.
• Prompt Engineering Still Crucial: The choice of prompt used during ORPO training influenced the final accuracy, although the gains from fine-tuning were large regardless.
• Improved Understanding: Post-SFT analysis showed the models became much better at differentiating similar classes and understanding context/intent, moving beyond simple keyword matching. Some issues remained (like TGU overprediction), suggesting areas for further refinement (class definitions, handling imbalance, perhaps adding synthetic data).
• Hyperparameter Sensitivity: Learning rate and steps influenced results, with diminishing returns observed when increasing steps from 100 to 200 for Gemma.

Key Takeaways

1. Accessible Fine-Tuning: Unsloth truly democratizes fine-tuning. By optimizing speed and memory (especially via quantization), it makes adapting powerful models feasible even on free-tier GPUs.
2. Efficient Adaptation: LoRA enables targeted fine-tuning, saving significant compute compared to full model retraining. A rank of 16 proved effective here.
3. Streamlined Alignment: ORPO offers a powerful, simplified approach combining SFT and preference alignment, yielding large performance improvements quickly and effectively.
4. Practical Workflow: This Unsloth + LoRA + ORPO combination provides a practical, efficient workflow for adapting LLMs to specific tasks like text classification, moving well beyond baseline prompt engineering performance.

Conclusion

By strategically combining the Unsloth library for speed and memory optimization, LoRA for parameter-efficient adaptation, and ORPO for streamlined fine-tuning, we successfully enhanced travel question classification accuracy for both Phi-3.5-mini and Gemma-2–9B models. Accuracy leaped from baseline levels of ~60–70% to impressive peaks of 82.86% (Phi) and 87.71% (Gemma) after just 100–200 steps of ORPO training on a single T4 GPU.

This demonstrates a powerful and accessible pathway for achieving high performance on specialized tasks, even within significant resource constraints. It underscores the importance of both efficient tooling (Unsloth) and advanced fine-tuning techniques (LoRA, ORPO) in unlocking the full potential of modern LLMs.

Ready to supercharge your own LLM fine-tuning? Explore the complete in-depth code in my Github and tools:

• Github: https://github.com/LupraV/TravelQA_Classification_LLM-SFT
• Unsloth: https://github.com/unslothai/unsloth
• TRL Library (ORPO): https://huggingface.co/docs/trl/
• ORPO Paper: https://arxiv.org/abs/2403.07691