Fine-Tuning Gemma-2B for Regional-to-English Translation Using LoRA & BitsAndBytes

Introduction

Building high-quality machine translation models for low-resource languages like Telugu remains a significant challenge in natural language processing. In this project, we fine-tune Google’s Gemma-2B model for Telugu → English translation using LoRA (Low-Rank Adaptation) and BitsAndBytes 4-bit quantization.

The goal is to efficiently adapt a large language model to a specific translation task while keeping hardware requirements minimal. We also integrate extensive dataset visualization to understand linguistic patterns before training.

We evaluate the fine-tuned model using BLEU, ROUGE, and METEOR scores to assess precision, recall, and balanced performance.

Purpose of the Model

The model translates Telugu sentences into English with improved accuracy by leveraging domain-specific datasets. This is particularly useful for:

• Multilingual customer support
• Educational resources translation
• Content localization for Telugu-speaking audiences

Why LoRA with BitsAndBytes?

We chose this combination because:

1. 4-bit Quantization (BitsAndBytes)

• Reduces VRAM usage, allowing the fine-tuning of large models on consumer GPUs like NVIDIA RTX 4090.
• Speeds up training without sacrificing significant performance.

2. LoRA (Low-Rank Adaptation)

• Updates only a small fraction of the model’s parameters.
• Enables parameter-efficient fine-tuning, making it possible to iterate quickly.

Model Training Steps

1. Base Model Loading

We start by loading Gemma-2B in 4-bit precision using BitsAndBytesConfig.

bnb_cfg = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)
mdl = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2b",
    quantization_config=bnb_cfg,
    device_map={"": 0}
)
tok = AutoTokenizer.from_pretrained("google/gemma-2b")

2. Dataset Preparation & Cleaning

We load the Telugu-Alpaca-Yahma dataset, clean unwanted characters, and select only the Telugu and English fields.

data = load_dataset("Telugu-LLM-Labs/telugu_alpaca_yahma_cleaned_filtered_romanized")

def clean_text(text):
    text = re.sub(r'[^\w\s.,!?]', '', text)
    text = re.sub(r'\s+', ' ', text)
    return text.strip().lower()

def select_fields(dat):
    return {
        "te": dat["telugu_instruction"].strip(),
        "en": clean_text(dat["instruction"])
    }

data = data.map(select_fields, remove_columns=data["train"].column_names)

We also visualized the dataset using:

• Line plots for text lengths
• Heatmaps for correlation between Telugu and English sentence lengths
• Word clouds for frequent term.
• Stacked bar charts, scatter plots, pie charts for data distribution

These visualizations ensured our training data was balanced and representative.

3. LoRA Fine-Tuning Configuration

We configure LoRA to modify key attention and projection layers for maximum impact on translation quality.

lora_cfg = LoraConfig(
    r=8,
    target_modules=["q_proj", "o_proj", "k_proj", "v_proj", "gate_proj", "up_proj", "down_proj"],
    task_type="CAUSAL_LM",
)
mdl = get_peft_model(mdl, lora_cfg)

4. Formatting Data for SFTTrainer

We create a prompt structure for consistent translation training.

def formatting_func(example):
    telugu_text = example['te'][0].strip()
    english_text = example['en'][0].strip().lower()
    return [f"translate the following telugu to english: {telugu_text} | english: {english_text} <eos>"]

5. Training

We use SFTTrainer for supervised fine-tuning.

sft_arguments = transformers.TrainingArguments(
    per_device_train_batch_size=12,
    gradient_accumulation_steps=8,
    warmup_steps=20,
    max_steps=100,
    learning_rate=1e-4,
    weight_decay=0.005,
    fp16=True,
    logging_steps=10,
    output_dir="outputs",
    optim="paged_adamw_8bit"
)

trainer = SFTTrainer(
    model=mdl,
    train_dataset=data["train"],
    args=sft_arguments,
    peft_config=lora_cfg,
    formatting_func=formatting_func,
)

trainer.train()

Evaluation Metrics

We tested the model on unseen Telugu sentences and compared its output to reference translations. Example:

Input: జీవితంలో సంతోషం పొందడం ఎలా? ఒక రెండు మాటల్లో చెప్పు.

Output: how can we be happy in life? give 2 sentences.

Metric Scores

BLEU~0.87 | METEOR~0.91 | ROUGE~0.90

• BLEU (Precision) — Captures exact word matches.
• ROUGE (Recall) — Measures coverage of correct phrases.
• METEOR (Balance) — Considers synonyms & semantic similarity.

Results

The model learned effective Telugu → English mapping, significantly improving translation accuracy. Training loss reduced from 2.34 → 0.08 over 100 steps, showing strong convergence.

Key Takeaways

• QLoRA + BitsAndBytes enables fine-tuning large models on a single RTX 4090.
• Dataset visualization prior to training improves awareness of language patterns.
• Model performs well on both short and long Telugu sentences.

Conclusion

This project demonstrates that with parameter-efficient fine-tuning and quantization, even large models can be adapted to low-resource languages without massive hardware.

Next steps:

• Extend to multi-language translation.
• Incorporate RLHF for better context alignment.

GitHub: https://github.com/revanthkumar1999/Gemma-multi-tasking-translation Hugging Face Model: https://huggingface.co/revanthkumar1999/gemma-2-Indian_languages-to-eng