Teaching an AI to Speak Naturally: Fine-Tuning a 1.3B Model for Conversational Sindhi-to-English Translation

🔗 Project Links

• Live Web App (Gradio): Sindhi English Translator
• Model Weights: Hugging Face Repository
• Dataset: Sindhi-English Parallel Corpus on Kaggle

1. The Problem: When AI Speaks Like a Textbook

Machine translation has made massive leaps with multilingual models like Meta’s NLLB-200 (No Language Left Behind). However, these generalized models suffer from a fundamental flaw: they are trained on web-crawled corpora consisting of highly formal text like Wikipedia articles, religious texts, and government documents.

While the base NLLB model excels at formal document translation, it completely breaks down when translating natural, daily-life conversational Sindhi into English. During my baseline evaluations, the model exhibited a tendency to produce stiff, literal “word salad” translations, drop entire clauses, or suffer from catastrophic hallucinations — literally generating archaic, biblical-sounding English when faced with simple Sindhi idioms.

ِExamples:

🟢 Original (Sindhi): نه نه مان پئسا ته نه وٺنديس. 🎯 Actual Result (English): No, no, I will not take the money. 🤖 nllb-1.3b Output: No, I don’t want money, so they won’t take it. 🚀 sindhi-nllb-1.3b Output: No, no, I will not take the money. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — 🟢 Original (Sindhi): پر پهريائين منهنجو پاسو ورتائين ته آءٌ اصل خوش ٿيس. 🎯 Actual Result (English): But when she took my side first, I was really happy. 🤖 nllb-1.3b Output: But first he took my side so I was really happy. 🚀 sindhi-nllb-1.3b Output: But when he first took my side, I was actually happy. — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

🟢 Original (Sindhi): سردار اٿيو ته هن جا ساٿي به سنوان ٿي اٿيا؛ سردار چيو، دوستو، مهرباني. 🎯 Actual Result (English): When the leader stood up, his companions also stood up straight; the leader said, ‘Friends, thank you.’ 🤖 nllb-1.3b Output: The chief said, “Friends, thank you”. 🚀 sindhi-nllb-1.3b Output: The Sardar got up, and his companions were also trying to get up; the Sardar said, ‘Friends, thank you.’

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

The objective of this project was clear: perform domain adaptation to train a specialized model capable of understanding the nuance, tone, and structure of spoken Sindhi and translating it into fluent, natural English.

2. Building the Dataset from Scratch

Conversational Sindhi datasets are practically non-existent. To solve this, I built a custom parallel corpus from scratch.

The data collection involved scanning 142 physical Sindhi books, specifically targeting novels and short stories rich in dialogue. The PDFs were converted into high-resolution images and preprocessed using OpenCV (applying CLAHE and Otsu thresholding for contrast enhancement). I then utilized Tesseract OCR (tesseract-ocr-snd) to extract the raw text.

To isolate conversational data from general narrative prose, I applied strict regex filtering, capturing sentences enclosed in quotation marks, sentences ending with question marks, and segments containing specific dialogue keywords:

import re

# Extracting and filtering conversational sentences from OCR
def is_conversational(sentence):
    if re.search(r'[“"”]', sentence): return True
    if '؟' in sentence or '?' in sentence: return True
    if sentence.startswith('-'): return True
    return False

conversation_keywords = [
    "چيو", "چيائين", "پڇيو", "وراڻيو", "ڳالهايو",
    "ڇا", "ڇو", "ڪٿي", "ڪيئن", "ڪڏهن", "ڪير",
    "مان", "تون", "توهان", "اسان", "منهنجو"
]

3. The Methodology

The pipeline spanned automated data preparation, rigorous cleaning, parameter-efficient fine-tuning, and full-stack deployment.

Phase A: Data Preparation via LLM

Raw OCR output is notoriously noisy. To generate high-quality parallel data, I engineered a prompt pipeline using Google’s gemini-3.1-flash-lite-preview. The LLM was tasked with fixing OCR typos, repairing dangling syntax, translating the cleaned Sindhi into natural English, and categorizing the sentence context (e.g., daily life, shopping, health). I implemented a fault-tolerant asynchronous loop to handle API rate limits and extract strictly formatted JSON objects.

# Gemini Data Preparation Prompt
prompt = PromptTemplate(
    template="""You are an expert Sindhi linguist building a Neural Machine Translation corpus.
    Process each of the following OCR-extracted Sindhi sentences to SALVAGE meaningful conversational data.

    Instructions:
    1. Fix obvious OCR typos.
    2. REPAIR DANGLING SENTENCES: Remove trailing conjunctions so the core sentence stands alone.
    3. Provide a highly accurate, natural English translation.
    4. Categorize the context strictly (e.g., greetings, family, daily_life, shopping).
    5. Output MUST be a raw JSON array of objects.

    Sentences to process:
    {sentences_chunk}
    """,
    input_variables=["sentences_chunk"]
)

Phase B: Data Exploration and Imbalance Handling

Visualizing the dataset revealed a severe class imbalance. My Exploratory Data Analysis (EDA) showed:

• The literature category dominated the dataset with nearly 25,000 sentences.

Number of Sentences by Category

• Crucial daily-life categories like greetings, shopping, and travel were severely underrepresented (fewer than 400 examples each).
• To prevent the model from becoming biased toward literary phrasing, I performed a stratified split and applied targeted upsampling to the minority classes, duplicating samples until each category reached a baseline of 500 instances.

from sklearn.utils import resample

minority_classes = ['greetings', 'shopping', 'travel', 'phone_conversation', 'education', 'health', 'work']
upsampled_dfs = []

for category in minority_classes:
    cat_df = train_df[train_df['category'] == category]
    # Upsample to 500 so the model pays attention to these contexts
    upsampled_cat = resample(cat_df, replace=True, n_samples=500, random_state=42)
    upsampled_dfs.append(upsampled_cat)

# Combine the majority classes with the newly upsampled minority classes
train_majority = train_df[~train_df['category'].isin(minority_classes)]
final_train_df = pd.concat([train_majority] + upsampled_dfs).sample(frac=1, random_state=42).reset_index(drop=True)

• Both languages followed an almost identical right-skewed distribution, confirming most conversational sentences are punchy phrases between 5 to 15 words long.

Distribution of Sentence Lengths

• A strong 1:1 word count correlation proved the LLM translations were direct and aligned.

Word Count Correlation

There is structural differences in how people talk depending on context.

Sindhi Sentence Length Across Different Categories

• Notice how the box for greetings is completely squished at the bottom? That confirms greetings are almost always 2 to 5 words (e.g., “Hello, how are you?”).
• Conversely, categories like work, literature, and family etc., have much taller boxes and longer whiskers, indicating complex, multi-clause sentences.

Phase C: Data Cleaning and Unicode Ghosts

I applied strict programmatic quality control: dropping invalid rows, removing duplicates, and auditing for “Unicode Ghosts” — rogue characters from Arabic and Urdu that visually resemble Sindhi characters but possess different Unicode values. Fixing this was vital to prevent the tokenizer from fracturing words during training.

# Curing Unicode Ghosts
replacements = {'ی': 'ي', 'ك': 'ڪ', 'ہ': 'ه'}

for bad_char, good_char in replacements.items():
    count = df['sd'].str.contains(bad_char, na=False).sum()
    if count > 0:
        df['sd'] = df['sd'].str.replace(bad_char, good_char)

4. The Model: Fine-Tuning Meta’s NLLB-1.3B

The foundational architecture is Facebook’s NLLB-200 1.3B, a massive Transformer-based Encoder-Decoder model. Because a 1.3B parameter model requires approximately 5.5GB of VRAM just to load, a full-parameter update on a standard GPU was impossible.

Instead, I utilized Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA). I targeted the core attention and feed-forward modules (q_proj, k_proj, v_proj, out_proj, fc1, fc2) with a Rank (r) of 32 and an Alpha of 64. This allowed the model to learn complex syntactic shifts without catastrophic forgetting.

from peft import LoraConfig, TaskType

# LoRA Configuration
peft_config = LoraConfig(
    task_type=TaskType.SEQ_2_SEQ_LM,
    inference_mode=False,
    r=32,
    lora_alpha=64,
    lora_dropout=0.05,
    target_modules=["q_proj", "k_proj", "v_proj", "out_proj", "fc1", "fc2"]
)

Training vs. Validation Loss

BLEU Score for Validation Data Across Training

Note: During training, Epoch 3 showed the smallest validation loss, but a significant rise in the BLEU Score occurred at Epoch 4 (53.32). I kept the weights at Epoch 4 as the final finetuned model.

5. Deployment on Gradio

Finally, we deployed the inference engine using Gradio, featuring a responsive, mobile-friendly UI with custom CSS.

Translation Hub

About the Model

6. Results & Accuracy

Evaluated on a holdout test set of conversational sentences, the results were night and day:

• Base Model (NLLB-200–1.3B) on test set: 26.74
• Fine-Tuned Model on test set: 53.90
• Fine-Tuned Model on training set: 70.94

This represents an astronomical improvement of +27.16 SacreBLEU points. The training score of 70.94 versus the test score of 53.90 proves the model generalized effectively without severe overfitting.

Qualitatively, the improvements in structural integrity were massive. When faced with complex emotional syntax (“هن کيس مصنوعي ڪاوڙ مان ڏٺو.”), the base model generated the stiff phrase “She saw him with artificial wrath.” The fine-tuned model correctly captured the human element: “He looked at him with artificial anger.” Most importantly, the fine-tuned adapter completely eliminated the base model’s tendency to hallucinate.

7. Limitations & Challenges

Engineering this pipeline came with real-world constraints:

• OCR & Data Scarcity: Tesseract OCR struggled heavily with the Arabic-Sindhi script. Extracting text required aggressive image preprocessing and automated loops to gather legible text at scale.
• API Bottlenecks: Generating parallel ground-truth data via the Gemini API resulted in frequent 503 errors due to server demand, forcing data synthesis to run exclusively during off-peak hours.
• Hardware Limits: OOM errors on standard T4 GPUs restricted the methodology strictly to PEFT/LoRA, preventing deeper foundational layer updates.

8. Conclusion & Future Scope

This project successfully bridged the gap between highly formal, web-crawled machine translation and the reality of daily spoken Sindhi. By engineering a comprehensive pipeline from OCR extraction to LoRA fine-tuning, I transformed a generalized 1.3B parameter model into a highly fluent, domain-specific translator.

What’s Next?

• Expansion of Digital Contexts: Integrating datasets from social media and WhatsApp to capture evolving internet slang.
• Idiomatic Edge Cases: Implementing targeted training pairs to handle cultural double-meanings (e.g., ensuring “Maya” translates contextually to “wealth” rather than just a proper noun).
• User Feedback Integration: Adding a real-time feedback mechanism to the Gradio interface for continuous data harvesting and reinforcement learning.

Final Thoughts:

This project was built to be open and accessible. If you are an NLP enthusiast, a linguist, or a native Sindhi speaker, I’d love for you to test the model, find its breaking points, and share your feedback. Your edge cases are exactly what will drive Version 2.0.

Explore the Project:

• 🚀 Try the Live App: Sindhi-English Translator
• 🧠 Explore the Model Weights: Hugging Face Repository
• 📊 Dive into the Dataset: Kaggle Parallel Corpus

If you’re working on similar domain adaptation problems, facing data scarcity in your native language, or just want to talk about LLMs and NLP pipelines, drop a comment below or connect with me! Let’s keep building AI that understands how people actually talk.