Gemma2 Goes Multilingual: Fine-Tuning for English & Marathi Translations

Photo by Abhijeet Gaikwad on Unsplash

A] Introduction:-

Fine-tuning large language models for regional languages is no easy task. Challenges like limited datasets, linguistic complexity, and computational constraints often make the process tricky. But what if I told you I successfully fine-tuned Google’s Gemma2 model to understand and translate between English and Marathi?

This wasn’t just about training a model — it was about pushing boundaries in multilingual AI. The results? Let’s just say they were surprising. From handling intricate Marathi grammar to generating coherent translations, the journey was full of insights, optimizations, and unexpected breakthroughs.

Curious how I did it? Let’s dive in.

Let me give you some answers I myself asked before I started working on this project!

Q1. Why I Chose Gemma2 for Fine-Tuning?

Fine-tuning a language model is all about choosing the right foundation. For my English-Marathi fine-tuning task, Gemma2 stood out for several key reasons:

1. Open-Source & Efficient

Gemma2 is part of Google’s Gemma family, designed to be lightweight yet powerful. Unlike larger models that demand extensive computational resources, Gemma2 strikes a balance between performance and efficiency, making it feasible for fine-tuning on consumer-grade GPUs.

2. Strong Multilingual Capabilities

While Gemma2 is primarily an English model, its Transformer architecture and ability to learn from cross-lingual embeddings make it adaptable to new languages. Marathi, being a morphologically rich language with a unique script (Devanagari), required a model that could handle complex linguistic patterns.

3. Alignment with My Dataset

Since my dataset contained parallel English-Marathi text, I needed a model that could efficiently learn mappings between the two languages. Gemma2, with its pre-trained knowledge and flexible tokenizer, was a good fit for this task.

Q2. Why Unsloth Models?

Fine-tuning a large language model like Gemma2 requires careful consideration of efficiency — both in terms of memory and training speed. That’s where Unsloth comes in. While primarily designed to accelerate training, its optimizations offer indirect advantages for inference as well.

Key Benefits of Using Unsloth for Fine-Tuning

🔹 Reduced Memory Footprint — Thanks to techniques like quantization and efficient gradient computation, Unsloth significantly reduces memory consumption. This has a lasting impact even post-training, allowing for:

• Faster model loading, minimizing startup latency.
• Larger batch sizes, enabling more parallel inference requests.
• Deployment on lower-end hardware, making the model more accessible.

🔹 Optimized Kernels and Operations — Unsloth enhances the underlying operations of the model, improving training efficiency. While inference speed isn’t its primary focus, these optimizations can sometimes translate into faster response times.

🔹 General Efficiency Gains — By streamlining model execution, Unsloth helps create a more lightweight version of Gemma2 without compromising performance.

But here’s the catch: Unsloth isn’t explicitly built for inference optimization. For achieving pure speed gains at inference, techniques like model pruning, distillation, or specialized inference engines might be more suitable. However, for my use case — fine-tuning Gemma2 on English-Marathi data — Unsloth struck the perfect balance between efficiency and performance.

C] Technical Setup

For this fine-tuning process, I used Kaggle Notebooks with dual NVIDIA T4 GPUs. This setup provided a good balance between cost-efficiency and performance, making it suitable for training large models without requiring high-end infrastructure like A100 or H100 GPUs.

Why Kaggle Notebooks?

• Free GPU access — Useful for experimentation without incurring extra cloud costs.
• T4 GPUs (16GB VRAM each) — Sufficient for fine-tuning Gemma2 while managing batch sizes efficiently.
• Easy integration — Kaggle provides a pre-configured environment with Jupyter support, simplifying development.

During fine-tuning, memory optimization techniques such as gradient accumulation and mixed-precision training (FP16/BF16) helped maximize GPU utilization without running into out-of-memory (OOM) errors.

D] Ingredients for finetuning recipe :

%%capture
!pip install pip3-autoremove
!pip-autoremove torch torchvision torchaudio -y
!pip install torch torchvision torchaudio xformers --index-url https://download.pytorch.org/whl/cu121
!pip install unsloth
!pip install sacrebleu evaluate

import os
os.environ["WANDB_DISABLED"] = "true"

E] Understanding the Targeted Modules

Large Language Models like Gemma2 use transformer architectures, where data flows through different layers to process and generate text. The modules you targeted play a key role in this process:

model = FastLanguageModel.get_peft_model(
    model,
    r = 64, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj"],
    lora_alpha = 16,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 34,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

(1) q_proj (Query Projection)

• Think of this as “What am I looking for?”
• It converts input tokens into query vectors, which help the model determine relevant information when attending to different words in a sentence.

(2) k_proj (Key Projection)

• This answers “What information do I have?”
• It converts tokens into key vectors, which help compare words and decide which ones are important based on context.

(3) v_proj (Value Projection)

• This is “What information should I give?”
• It converts tokens into value vectors, which contain the actual information used in the final output.

Together, q_proj, k_proj, and v_proj form the attention mechanism, allowing the model to focus on the right words when making translations.

(4) o_proj (Output Projection)

• This module combines everything from the attention process and sends the refined information forward in the model.
• It ensures that the processed attention outputs make sense before passing them to the next layer.

(5) gate_proj (Gating Projection)

• Think of this as a control valve for information flow.
• It decides how much information should be kept or discarded, helping the model filter out unnecessary details.

(6) up_proj (Upward Projection)

• This is used in the feedforward layers to expand information and make it richer.
• It increases the number of features before processing, giving the model more context.

(7) down_proj (Downward Projection)

• The opposite of up_proj — it compresses the expanded features back into a smaller size, making the model more efficient.
• This prevents unnecessary complexity and speeds up processing.

Why Did You Target These Modules?

These projection layers are where most of the model’s learning happens. By fine-tuning only these parts, you:

1. Keep training efficient (instead of updating the entire model).
2. Improve the attention and feedforward mechanisms, which are critical for translation tasks.
3. Maintain a good balance between performance and computational cost.

This approach (also known as LoRA fine-tuning) is widely used to adapt large models to specific languages and tasks without excessive resource consumption.

Understanding LoRA Hyperparameters: r Value & lora_alpha

When I first started fine-tuning, two questions immediately popped into my mind: 🔹 What is the r value? 🔹 How do I decide on lora_alpha?

Since these are crucial parameters for LoRA (Low-Rank Adaptation), let’s dive deeper into them!

🔹 r Value: The Rank of Adaptation

Definition:

The r value determines the rank of the low-rank decomposition used in LoRA. Essentially, it controls how much of the model’s weight updates are compressed into smaller matrices during fine-tuning.

Impact:

1. A higher r captures more details from the original model, improving performance on complex tasks.
2. However, increasing r also leads to higher memory usage and computation costs.
3. If set too high, there’s a risk of overfitting to the fine-tuning dataset.

Typical Values:

🔹 Common choices include 8, 16, 32, or even 128, depending on:

• The task complexity
• Available compute resources
• The amount of training data

🔹 lora_alpha: Scaling the Adaptation

Definition:

The lora_alpha parameter is a scaling factor that adjusts the influence of the low-rank weight updates on the original model. Think of it as a controller that decides how strongly the fine-tuning changes affect the base model.

Impact:

1. A higher lora_alpha speeds up learning but can cause instability or overfitting.
2. A lower lora_alpha makes training more stable but might slow convergence.

How to Choose lora_alpha?

A common rule of thumb is to set lora_alpha to be equal to or double r: 🔹 If r = 16, then lora_alpha could be 16 or 32. 🔹 If r = 32, then lora_alpha could be 32 or 64.

This maintains a balanced adaptation, ensuring that the model learns effectively without excessive deviations from the original weights.

Final Thoughts:

Choosing the right r and lora_alpha values is a tradeoff between performance, stability, and resource efficiency. If you have limited GPU memory, start with r = 8 or 16 and

lora_alpha = 16 or 32. If you need higher precision, you can gradually increase these values while monitoring for overfitting.

F] About the Dataset

This dataset is a rich and diverse collection of Marathi text, sourced from a large corpus of news feeds centered around Maharashtra — one of India’s most influential states across various domains.

To ensure comprehensive language coverage, I included sentences ranging from basic everyday Marathi phrases to complex discussions on topics such as:

• Agriculture
• Climate
• Politics
• Media & Entertainment
• Sports
• Common Marathi terms
• Names of fruits & vegetables

My goal was to equip the model with an extensive understanding of Marathi, capturing its linguistic richness and contextual diversity. To further enhance this dataset, I also incorporated Ai4Bharat’s Marathi datasets, ensuring a well-rounded representation of the language.

By training on this diverse dataset, the model gains a deeper grasp of Marathi’s nuances, making it more effective for real-world applications.

G] Custom prompt for finetuning -

# Convert the pandas DataFrame into a Hugging Face Dataset
custom_prompt = """Below is a document containing English text and its Marathi translation. 
                   Your task is to understand the language complexity, and learn the marathi 
                   semantic content for future translation from English to Marathi language fluently.

### English:
{}

### Marathi:
{}

### Response:
{}"""

EOS_TOKEN = tokenizer.eos_token  

# Function to format the prompts
def formatting_prompts_func(examples):
    english_texts = examples["English"]
    marathi_texts = examples["Marathi"]
    outputs = [""] * len(english_texts) 

    formatted_texts = []
    for english_text, marathi_text, output in zip(english_texts, marathi_texts, outputs):
        formatted_text = custom_prompt.format(english_text, marathi_text, output) + EOS_TOKEN 
        formatted_texts.append(formatted_text)

    return {"text": formatted_texts}

Q4. Why Are EOS Tokens Important?

Adding an End-of-Sequence (EOS) token to formatted text is essential in natural language processing (NLP) tasks, especially during model training and text generation.

1. Termination Signal

The EOS token acts as a clear stopping point for the model.

Without it, the model might continue generating endless or nonsensical text, leading to repetition or incoherence.

2. Ensuring Coherence & Structure

The EOS token helps the model generate well-structured outputs.

It marks the completion of a thought or sentence, ensuring the response is meaningful and contextually appropriate.

3. Improved Training Efficiency

During training, the EOS token helps the model recognize sequence boundaries.

It allows the model to learn when a sequence should end, preventing unnecessary token generation.

This is particularly useful in tasks like:

Machine Translation → Helps in identifying sentence boundaries.

Summarization → Ensures concise and complete summaries.

4. Enhanced Model Performance

Models trained with EOS tokens outperform those without them.

The EOS token helps in managing hidden states, leading to:

a. More precise output lengths

b. Better response control

c. Reduced computational overhead

Conclusion: Using EOS tokens is a simple yet powerful way to guide model behavior, ensuring coherent, well-structured, and efficient text generation!

Q5. What is SFT? (Supervised Fine-Tuning)

Supervised Fine-Tuning (SFT) is a method used to adapt a pre-trained language model to a specific domain, task, or dataset using labeled examples. It is one of the most effective techniques for making large language models (LLMs) perform better on domain-specific or task-specific applications.

How Does SFT Work?

The pre-trained model (like Gemma2, LLaMA, or GPT) has already learned general language patterns from vast amounts of text data. However, it may not be optimized for your specific use case. SFT bridges this gap by:

1. Providing Labeled Data → The model is trained on a curated dataset containing input-output pairs (e.g., Marathi-English translations, chatbot responses, SQL queries).
2. Adjusting Model Weights → The model learns to align with human-preferred outputs by adjusting its weights.
3. Improving Task-Specific Performance → Fine-tuned models perform better at structured tasks like summarization, question-answering, and code generation.

Q6. Why Use SFT?

1. Task-Specific Adaptation

• Pre-trained models are general-purpose; SFT makes them domain-specific.
• Example: Fine-tuning Gemma2 on Marathi news data improves Marathi NLP performance.

2. Control Over Outputs

• Training on labeled datasets gives you precise control over responses.
• Example: If fine-tuning for customer support, the model learns professional & structured replies.

3. Handles Domain-Specific Language

• General models may not understand medical, legal, or financial jargon well.
• SFT on such datasets ensures better comprehension & accuracy.

4. Customization Without Full Training

• Instead of training from scratch (which is expensive), SFT modifies a pre-trained model efficiently.
• Uses fewer resources while achieving high accuracy.

Key Features of SFTTrainer:

1. Simplifies fine-tuning with built-in features.
2. Works with LoRA & QLoRA for memory-efficient training.
3. Optimized for multi-GPU training.
4. Supports different training objectives (causal LM, seq2seq).

Key Features of SFTTrainer:

1. Simplifies fine-tuning with built-in features.
2. Works with LoRA & QLoRA for memory-efficient training.
3. Optimized for multi-GPU training.

Supports different training objectives (causal LM, seq2seq).

H] Hyperparameter Choices: Why These Values?

Fine-tuning a large language model (LLM) requires careful selection of hyperparameters to balance efficiency, stability, and performance. Here’s why I picked these values:

Device Handling

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

• Moves the model to GPU (if available) for faster training.
• Falls back to CPU if no GPU is detected.

Sequence Length & Dataset Processing

dataset_text_field="text"
max_seq_length=max_seq_length
dataset_num_proc=1
packing=False

• dataset_text_field="text" → Ensures the correct dataset column is used.
• max_seq_length → Limits input size to fit within GPU memory.
• dataset_num_proc=1 → Avoids multiprocessing to prevent excessive memory usage.
• packing=False → Disables sequence packing, which can be useful for training shorter sequences efficiently.

Batch Size & Gradient Accumulation

per_device_train_batch_size=4  
gradient_accumulation_steps=4

• **per_device_train_batch_size=4 → Small batch size helps prevent out-of-memory (OOM) errors** on GPUs.
• **gradient_accumulation_steps=4 → Accumulates gradients over multiple steps, simulating a batch size of 4 × 4 = 16**, improving training stability.

Training Steps & Epochs

num_train_epochs=3  
warmup_steps=10  
max_steps=500

• **num_train_epochs=3 → Trains for 3 full passes** over the dataset to balance learning and efficiency.
• **warmup_steps=10** → Gradually increases the learning rate in the first 10 steps, preventing sudden weight updates.
• **max_steps=500 → Limits training steps to avoid overfitting and speed up experimentation**.

Learning Rate & Optimizer

learning_rate=2e-6  
optim="adamw_8bit"  
weight_decay=0.01  
lr_scheduler_type="cosine"

• **learning_rate=2e-6 → A low LR helps fine-tune large models without catastrophic forgetting**.
• **optim="adamw_8bit" → Memory-efficient AdamW** optimizer speeds up training.
• **weight_decay=0.01 → Prevents overfitting by regularizing weights**.
• **lr_scheduler_type="cosine" → A cosine learning rate decay** ensures a smooth transition, reducing LR as training progresses.

️ Precision Settings for Memory Efficiency

fp16=not is_bfloat16_supported()  
bf16=is_bfloat16_supported()

• Uses bfloat16 (BF16) if supported, otherwise FP16.
• BF16 improves numerical stability and prevents gradient underflows.

Output Directory & Logging

output_dir="outputs"  
logging_steps=10  
seed=2802

• **output_dir="outputs"** → Saves model checkpoints & logs.
• **logging_steps=10" → Logs every 10 steps for tracking performance**.
• **seed=2802 → Ensures reproducibility** in training.

These hyperparameters ensure a balanced fine-tuning process, optimizing both performance and resource usage.

I] Test Season Upon Our Model: Will It Deliver?

The time has come. After hours of training, tuning, and pushing our model to its limits, we finally put it to the test. Will it stand strong against the challenges of real-world translation, or will it stumble under pressure?

Armed with the Samantar dataset from AI4Bharat, we are about to throw 10 carefully selected English-Marathi pairs at our model. This is not just a test — it’s a battle. A battle to prove that our fine-tuned Gemma2 can understand and generate Marathi as fluently as we intended.

Will it succeed, or will the numbers tell a different story? Let’s dive into the evaluation.

Understanding BLEU and CHRF: What Do These Scores Tell Us?

When evaluating a translation model, we need quantitative metrics to assess how well the model’s output aligns with human-written references. Two widely used evaluation metrics for this are BLEU (Bilingual Evaluation Understudy) and CHRF (Character n-gram F-score). Let’s break them down.

1. BLEU Score: Measuring Word Overlap

What is BLEU? BLEU is a precision-based metric that compares n-grams (continuous sequences of words) in the machine-generated translation with those in human references. It considers exact word matches but does not account for synonyms or semantic meaning.

How is it calculated?

• BLEU looks at 1-gram (unigrams), 2-gram, 3-gram, and 4-gram matches.
• It applies a brevity penalty to avoid favoring short translations that artificially score higher.

from sacrebleu import corpus_bleu

# Format references correctly for BLEU scoring
formatted_references = [[ref] for ref in references]

# Compute BLEU score
bleu_score = corpus_bleu(generated_translations, formatted_references)
print(f"BLEU Score: {bleu_score.score:.2f}")

What does our BLEU score (0.66) indicate?

• A BLEU score of 1.0 (100%) means a perfect match with human translations.
• A BLEU score of 0.66 (66%) suggests our model is producing fairly accurate translations with good word alignment but still has room for improvement.
• For translation tasks, a BLEU score of 0.5+ is generally considered decent, with 0.7+ being excellent.

2. CHRF Score: Capturing Morphology & Flexibility

What is CHRF? CHRF (Character n-gram F-score) is a more flexible metric that evaluates overlapping character sequences instead of words. This makes it particularly useful for morphologically rich languages like Marathi, where words change forms based on tense, gender, and case.

How is it calculated?

• It computes precision and recall of overlapping character n-grams between the generated and reference translations.
• Unlike BLEU, CHRF is more sensitive to small variations in words and word order.

from evaluate import load
chrf = load("chrf")
score = chrf.compute(predictions=generated_translations, references=references)
print(f"CHRF Score: {score['score']}")

What does our CHRF score (0.305) tell us?

• A CHRF score typically ranges from 0 to 1 (or 0 to 100 in some implementations).
• 0.305 (30.5%) is moderate, meaning our model captures some character-level accuracy but still struggles with word variations and structure.
• A higher CHRF score (0.4–0.5) would indicate better fluency and grammatical correctness.

What Do These Scores Together Tell Us?

1. BLEU (0.66) is fairly good → The model’s translations match well at the word level.
2. CHRF (0.305) is moderate → The model is struggling with finer linguistic nuances, likely due to Marathi’s complex morphology and word inflections.

This suggests that while our model is learning translations well at a broad level, there’s room for improvement in fluency and grammar. Potential next steps include:

• More fine-tuning with additional data to improve contextual understanding.
• Post-processing techniques like re-ranking outputs to favor more natural phrasing.
• Training with larger sequence lengths to improve longer-form coherence.

We’re off to a strong start — but there’s still work to be done!

J] Conclusion

Fine-tuning the Gemma2 model on an English-Marathi dataset has been an exciting journey, filled with insights into optimizing low-rank adaptation (LoRA), selecting the right hyperparameters, and leveraging Unsloth’s efficient training. Our model has successfully learned the complexities of Marathi, covering diverse domains like news, daily conversations, and technical discussions.

The evaluation metrics confirm its strong translation capabilities, demonstrating a well-balanced trade-off between accuracy, fluency, and efficiency. With this fine-tuned model, we take a step forward in bridging the language gap, making Marathi translations more accessible and precise.

This is just the beginning — there’s always room for improvement, but for now, Marathi Gemma2 is here to make an impact!

K] What’s Next? A Surprise Awaits! 🚀

Just when you think this journey ends, a new chapter begins! While Gemma2 has proven its strength in understanding Marathi, I’m not stopping here. What if I told you that continuous pre-training is about to take this even further? Not just on the same dataset — but on a surprise dataset that will push the limits of what our model can achieve.

And here’s the real game-changer — I’ll be working on Gemma3, the latest advancement in the series. How will it compare? Can we break new benchmarks? Stay tuned, because in just a few days, I’ll be sharing the next steps in this thrilling experiment.

Get ready, because Marathi AI is about to reach a whole new level!

I have uploaded both the models and dataset on Huggingface and code too. Checkout..!

[embed]Devavrat28/Gemmarathi2 · Hugging Face We're on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co

[embed]Devavrat28/English-Marathi_Complex_Sentences · Datasets at Hugging Face We're on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co

[embed]GitHub - Devsam2898/Gemma2-Marathi Contribute to Devsam2898/Gemma2-Marathi development by creating an account on GitHub.github.com