Applying Smarter Training and Deeper Optimization on Hindi to Kangri Translation with NLLB-200 600m Model

In the previous part of this series, we took the first steps in adapting Meta’s NLLB-200 multilingual model to support Hindi → Kangri translation. By customizing the tokenizer, fine-tuning on aligned sentence pairs, and incorporating Kangri into the model’s language space, we were able to build a strong baseline — one of the first practical machine translation systems for this underrepresented Himalayan language.

In this part, we dive into the more advanced techniques used to push the performance boundaries further. From Step Training, label smoothing and beam search decoding, to training with mixed precision and smarter scheduling, we explore how small design choices translate into meaningful improvements.

Timeline

In our previous version of model, we had 149k lines of kangri corpus text. And about 20k sentence pairs of Hindi to kangri. The BLEU score of v1 is 14.87[this time]. Previously due to low data I was not able to test the model fully. But this time we had about 5k pairs in our test dataset. It is a product of multiple trained, else the final model is the product of multiple trainings with different techniques, applied layer by layer.

v2 — v1 model with Enhanced Tokenizer + 20k new training Pairs

score: v2 (BLEU: 18.00)

• Foundation: Started from the model in the first blog — trained on:
• 149k raw Kangri corpus
• 18k Hindi–Kangri sentence pairs
• Tokenizer Rebuilt:
• Used 270K Kangri corpus to train a new SentencePiece tokenizer increased the vocab_size with about 57k new words
• Captured Kangri morphology more effectively
• Fine-tuned with an additional new 20k pairs
• Trained for 1 epoch using Hugging Face’s Trainer class function on Kaggle (2× T4 GPUs)

Result:

• BLEU: 18.00
• Readable output with better Kangri-specific subword segmentation
• Still lacked fluency, variation, and generalization

v2.1 — Infrastructure Upgrade (Accelerate, Dataloader)

Not benchmarked separately; improvements merged into v3

• Switched to custom training loop using Accelerate:
• Full control over mixed-precision (FP16)
• Faster training and memory-efficient batches
• Used 49k aligned sentence pairs (8:1:1 split)
• Introduced structured CSV logging and experiment tracking

Result:

• Not formally benchmarked due to memory issues and early OOMs
• Functioned as a bridge to v3, where performance tuning began

v3 — Performance Tuning Phase

score: v3 (BLEU: 22.54)

• Introduced key optimization techniques:
• Label Smoothing (ε = 0.1) >>>slightly reduce the confidence of the “correct” label and distribute some probability mass to the incorrect labels:
• Beam Search (num_beams=4, no_repeat_ngram=3, repetition_penalty=1.2)
• num_beamsHow many sequences to explore,Improves translation fluency
• no_repeat_ngram_sizePrevents repeated phrases , Stops looping phrases
• repetition_penaltyPenalizes token reuse
• Cosine LR Scheduler>>Start fast, slow down gently.
• Continued FP16 training>>model limited to FP 16 length, saves memory without losing much accuracy
• First experiments with lightweight augmentation and decoding control

Result:

• BLEU: 22.54
• Loss: 1.74 after epoch 1
• Fluent, less repetitive outputs
• Confident with rare and long sequences

v4 — Robustness with Data Augmentation + Multi-Epoch

score: v4 (BLEU: 24.39)

• Introduced custom Data Augmenter pipeline:
• Synonym Replacement (बहुत → काफी, अच्छा → उत्तम)
• Random Insertion (भी, तो, ही added)
• Random Deletion (light word removal)
• Augmentation was applied dynamically during training
• Switched to Kaggle P100 GPU (16GB) for 3-epoch fine-tuning
• Checkpoints manually pushed to Hugging Face Hub after each epoch

Result:

• BLEU: 24.39
• CHRF++: 52.70
• Stable training across sessions
• More robust and diverse translations

v5 — Dropout Regularization (0.3)

score: v5 (BLEU: 23.15)

• Introduced dropout = 0.3 to mitigate early overfitting from v4
• Dropout applied to encoder and decoder hidden states
• Continued with v4 rest.

Result:

• BLEU: 23.15
• Slight reduction in training loss
• More generalization on unseen patterns
• Small tradeoff in BLEU due to stronger regularization

v6 — Final Tuning with Dropout 0.1

score: v6 (BLEU: 26.04 — Best)

• Decreased dropout to 0.1 lower regularization
• Targeted token hallucinations and rigid decoding
• Trained on same augmented dataset using the step-by-step resume method

Combined:

• Label Smoothing
• Augmentation (Hindi & Kangri sides)
• Beam Search + Constraints
• Cosine LR
• Mixed Precision (FP16)

Result:

• BLEU: 26.04 (Best)
• CHRF++: 53.93
• BERTScore-F1: 0.933
• More flexible word order, less repetition
• Ready for real-world deployment

Results

Some Important Hyperparameters to remember

Data Pipeline: Data Augmentation

I built a custom augmenter that generates new sentences by tweaking existing ones. It’s like giving the model slightly rephrased questions — so it learns the concept, not just the answer.

What does it do?

• Synonym Replacement: “बहुत” becomes “काफी” or “अत्यधिक”.
• Random Insertion: Adds “तो”, “ही” to make it feel more like speech.
• Random Deletion: Drops unimportant words.

Maths Behind It

Let’s say the model learns a function f(x) where x is a Hindi sentence.

We create a slightly tweaked version: x~ = x + δ

• δ is a small change (like adding/removing a word).

Goal: make sure the model prediction doesn’t change much. f(x) ≈ f(x~)

This teaches the model to be stable even when users input noisy or slightly different sentences.

Math Behind Overfitting:

Let’s define:

• R_emp(f) = training loss (how well it fits your data)
• R(f) = true loss (how well it generalizes)

If R_emp(f) keeps dropping but R(f) goes flat — you’re overfitting.

We calculate:

• gap = val_loss - train_loss
• If gap > 0.1: big overfitting

Tokenization + Masking: Clean Inputs, Clean Gradients

NLLB needs special language codes: hin_Deva for Hindi and kang_Deva for Kangri.

We tokenize the input, then pad to fixed length. But here’s the trick:

• Pad token = <pad> = ID like 0
• We mask it using -100 so the model doesn’t compute loss on it.

Why?

Loss formula (cross entropy):

Cross-entropy measures how well the predicted probability distribution (from the model) matches the actual distribution (where the correct token has probability = 1).

• The more confident the model is in predicting the correct outcome, the lower the loss.
• The more confident the model is in predicting the wrong outcome, the higher the loss.

If we include pad tokens, they mess up the gradients. Masking ensures the loss is only computed on real words. To go more deep visit this article

Full Training Walkthrough: How We Fine-Tuned NLLB for Hindi → Kangri

In this part of the blog series, I’m going to walk you through the full training pipeline used to fine-tune Meta’s NLLB-200 model to understand and generate Kangri — a low-resource Himalayan language. But more than that, I’ll break it down like we’re doing it together in a college dorm room: math, intuition, real examples — and no fluff.

You already saw the results (BLEU 26.04!) — here’s how we got there.

Accelerator + Mixed Precision

accelerator = Accelerator(mixed_precision="fp16")

FP32 vs FP16: What’s the Real Difference?

When training deep learning models, numbers are everywhere — weights, gradients, activations. And these numbers are stored as floating point values.

What Does This Actually Mean?

• FP32 (float32) keeps up to ~7 decimal digits accurately. It’s like writing: 0.1234567
• FP16 (float16) keeps only ~3–4 decimal digits accurately. It’s like writing: 0.1234 and rounding the rest off.

So if your model is multiplying many values like 0.123456 × 0.987654, FP32 will give a more precise result, while FP16 might round off slightly.

Pros and Cons of FP16

Advantages:

1. Less Memory Usage: FP16 uses half the space. You can train larger models or use bigger batches. Example: FP32 uses 4 GB for your model, FP16 might use only 2 GB.

• Faster Training (on modern GPUs): New GPUs like NVIDIA T4 or A100 are optimized to perform FP16 calculations faster. This gives you shorter training times.

1. Lower Bandwidth: Faster data movement between memory and GPU cores.

Disadvantages:

1. Lower Precision: If numbers get very small (like 1e-7) or very large, FP16 may round them badly. This can cause instability during training if not handled carefully.
2. Gradient Underflow: If gradients become too small, they might just get rounded to 0 in FP16 — meaning no learning.
3. Needs Special Hardware Handling: On older GPUs, FP16 might not help — and can even be slower or less stable.

Solution: Mixed Precision (The Best of Both Worlds)

Instead of using only FP16, we use mixed precision:

• Use FP16 where it’s safe (activations, forward pass).
• Use FP32 for critical math (loss calculation, gradient updates, Weight Updates).

This is what the line in your code does:

accelerator = Accelerator(mixed_precision="fp16")

It handles all the tricky switching automatically — giving you the speed of FP16 without the headaches.

Tokenizer and Model Load

tokenizer = AutoTokenizer.from_pretrained(HF_MODEL)
model = AutoModelForSeq2SeqLM.from_pretrained(HF_MODEL)

What it does:

• The tokenizer turns your Hindi sentence like "यह अच्छा है" into token IDs like [50265, 12098, 715].
• The model is an encoder-decoder transformer. The encoder understands Hindi, the decoder writes in Kangri.

Mathematics (simplified):

• Explanation of variables:
• x: tokenized Hindi sentence (input)
• y_hat​: predicted Kangri sentence (output)

Training Dataloader

train_loader = DataLoader(..., batch_size=8, shuffle=True)

Why batch size matters:

• Too small (e.g. 1): noisy training, slow convergence
• Too big (e.g. 32 on small VRAM): out-of-memory (OOM)

We used batch size = 8, which is optimal with fp16 on P100(16GB).

Optimizer: AdamW

optimizer = AdamW(model.parameters(), lr=3e-5, weight_decay=0.01)

AdamW = Adam + Weight Decay

• Adapts learning rate per parameter
• Weight decay keeps weights from growing too big and memorizing stuff.

Formula:

Weight decay encourages the model to keep weights small, which:

• Reduces overfitting
• Improves generalization
• Helps avoid memorizing training data

Effect of Tuning Weight Decay λ

When λ = 0 (No Regularization)

• Effect: The model memorizes the training data too well
• Result: Low training loss, but high validation loss — classic overfitting.
• Why?: The weights grow freely without restriction.

When λ is too high (e.g., λ>0.05)

• Effect: The model becomes too cautious and doesn’t learn effectively.
• Result: Both training and validation losses stay high — underfitting.
• Why?: The optimizer keeps shrinking weights too aggressively, even when they’re needed to represent meaningful patterns.

Recommended:

• For fine-tuning transformer models like NLLB, a typical good value is:

This gently penalizes large weights and improves generalization without hurting performance.

Cosine Learning Rate Scheduler

lr_scheduler = get_scheduler(
    "cosine",
    optimizer=optimizer,
    num_warmup_steps=100,
    num_training_steps=steps
)

What it does: The learning rate starts high, stays flat during the warm-up phase, and then decays smoothly following a cosine curve. This helps the model learn quickly at first and then converge gradually.

Formula:

Example:

Why it’s useful:

• Smooth learning rate decay prevents sudden drops that can hurt convergence.
• Encourages fast early learning and fine-grained tuning at the end.
• Helps reduce overfitting and sharp loss fluctuations.

Training Loop — What Actually Happens

outputs = model(**batch)
loss = outputs.loss
accelerator.backward(loss)
optimizer.step()

• The model takes a batch of Hindi→Kangri data and predicts the output tokens.
• The loss is calculated by comparing the predicted sentence to the actual Kangri sentence.
• We use backpropagation to update the model’s internal weights so it learns better next time.

Loss Function: Cross Entropy

• The more confident the model is in predicting the correct outcome, the lower the loss.
• The more confident the model is in predicting the wrong outcome, the higher the loss.

Validation BLEU Score

bleu_score = bleu.compute(predictions=preds, references=[[l] for l in labels])["score"]

What it measures: BLEU (Bilingual Evaluation Understudy) is a precision-based metric for translation. It tells us how many n-grams (i.e., word chunks) in the predicted sentence also appear in the reference sentence.

• Score range: 0 (worst) to 100 (perfect match)
• Use case: A BLEU score of 25–30 is considered good for low-resource language models.

Formula (simplified):

Brevity Penalty (BP)

Benchmarks

8. Checkpointing

torch.save({
  "model": model.state_dict(),
  "optimizer": optimizer.state_dict(),
  "scheduler": scheduler.state_dict(),
  "best_bleu": bleu_score
}, ckpt_path)

• Save model + optimizer + scheduler + BLEU after every epoch
• Resume training exactly where it stopped if you crash

HF model link click here

Github Repo click here

Dataset click here

Final Thoughts

Fine-tuning isn’t just “press train and wait.” It’s a careful game of shaping the loss, balancing speed and stability, and watching overfitting like a hawk.

This training setup — with label smoothing, beam search, cosine scheduler, and proper checkpoints — gave us a BLEU score jump from 14.87 → 26.04.

And, In the future, If happend , we just need to train a model to kangri to hindi model. Whoa, we’ll have Bi-directional model. Or use hindi as middle lamguange and we can train the NLLB to do English to kangri. Any other languange in the world. Well Currently, i am interested in MCP thing. So, will there a v3 of NLLB? idk.