IndicParam — A Benchmark for Low-Resource Indian Languages

**tl;dr: **This is a quick summary of my observations on Gemini-2.5-Flash model’s performance on a recently released (publicly available) “IndicParam” benchmark released by BharatGen for 11(+1) low to extremely low-resource Indian languages.

**About the Benchmark:* This is how the dataset description reads on Huggingface: “IndicParam is a graduate-level benchmark designed to evaluate Large Language Models (LLMs) on their understanding of low- and extremely low-resource Indic languages. The dataset contains 13,207 multiple-choice questions (MCQs) across 11 Indic languages, plus a separate Sanskrit–English code-mixed set, all sourced from official UGC-NET language question papers and answer keys.”*

Languages Covered: Low-resource (4): Nepali, Gujarati, Marathi, Odia Extremely low-resource (7): Dogri, Maithili, Rajasthani, Sanskrit, Bodo, Santali, Konkani, Code-mixed: Sanskrit–English (Sans-Eng). Of these, Gujarathi, Odia, and Santali have their own scripts, and the rest are in Devnagari (or Devnagar-Eng for Sanskrit).

Summary of the paper’s reported results: In the paper, they report evaluations with 19 models, highest average performance being for GPT5 (45% accuracy), and highest per-language performance seems to be for the code-mixed Sanskrit-English data (64.6% accuracy as per Table 4 in the paper). Another important observation is Deepseek-3.2 seems to come quite close to GPT5, and even beats it for some languages.

What I did: Although there are 19 models, I found Gemini line of models missing, and in my experience with Telugu benchmarking (details here), Gemini models were way better than other contemporary proprietary models. So, I ran the benchmark on Gemini-2.5-Flash (cost reasons. This costed less than 1$ to run. Gemini3-Pro and the likes would cost 10 times more). I used Inspect-AI to run my eval and OpenRouter to call the API [code is shared at the end of the post]. It was zero-shot prompting, and a normal generate and evaluate style (i.e., no chain of thought, react prompting etc).

What I found: Overall accuracy is 11% higher with Gemini (~56%) than the GPT5 model which had the best accuracy reported in the paper (45%). Sanskrit-English code mixed data had the best accuracy among the language subsets, and Bodo had the lowest, which is consistent with the paper’s order, but there is a >10% difference in absolute numbers, again. Detailed language wise performance of gemini-2.5-flash is below:

{'Bodo': 45.23990860624524,
 'Dogri': 51.11976630963972,
 'Gujarati_surya': 51.81992337164751,
 'Konkani': 48.94578313253012,
 'Maithili': 52.02177293934681,
 'Marathi': 53.81526104417671,
 'Nepali': 52.408477842003855,
 'Oriya': 58.752166377816295,
 'Rajasthani': 62.94117647058823,
 'Sanskrit': 70.95057034220532,
 'Sanskrit Mix': 77.54891864057673,
 'Santali': 50.74455899198167}

What it means: I did not do any qualitative analysis, but to me, it is unsurprising to see Sanskrit and Sanskrit mix do way better than the rest. Considering the amount of classical Sanskrit texts available online with commentaries in multiple Indian languages, the LLMs must have all seen all that during their pre-training. It is interesting to see Rajasthani, which the authors categorized as “extremely low resource”, do much better than all the (just) “low resource” category languages: Nepali, Gujarati, Marathi and Odia. Is it because Rajasthani is somehow closer to the relatively high-resource Hindi compared to these other languages? It would have been nice to see further discussion on these kind of things in the paper. Overall, I think we should not ignore Gemini models when evaluating on Indian languages.

Bigger questions: This benchmark is a great starting point in understanding how LLMs do on many Indian languages. Ofcourse, there are bigger questions around what does performance on multiple-choice questions even mean in real world? What is, for example, the use case for Bodo speakers compared to people looking at Sanskrit when it comes to LLMs? Are they look at similar sort of things or no? For example, translation and analysis of classical texts may be a common use case for Sanskrit, not necessarily for Bodo. How does one estimate that? What do these differences mean when we evaluate? I think governments in multilingual countries should really spend more time on these kind of questions along with creating useful benchmarks like this.

General Caveats: Multiple choice questions are not necessarily the ideal format to test a model’s understanding of a language — they just provide a easy way to evaluation. The format is also quite sensitive and gamed easily, and researchers have been critical of that format in the recent past (e.g, Li et.al, 2024). So, the reality may be worser than these numbers.

Code for replication and running with other models. The output can be saved in json format which makes it easier to do any meta-analysis later.

from inspect_ai.dataset import Sample
from inspect_ai import Task, task
from inspect_ai.scorer import choice
from inspect_ai.solver import multiple_choice
from datasets import load_dataset

def make_dataset():
  ds = load_dataset("bharatgenai/IndicParam")["test"]
  dataset = []
  questions = ds['question_text']
  option_a, option_b, option_c, option_d = ds['option_a'], ds['option_b'], \
                                           ds['option_c'], ds['option_d']
  choices = [[option_a[i], option_b[i], option_c[i], option_d[i]] for i in range(len(option_a))]
  answers = ds['correct_answer']
  language = ds['subject']
  for i in range(0,len(questions)):
    metadata = {'language': language[i],
                }
    dataset.append(Sample(input=questions[i], choices=choices[i], target=answers[i], metadata=metadata))
  return dataset

#Define the evaluation task for inspect:
@task
def eval_indicparam():
    task_name = "IndicParam_AllLngs_Gemini"  #change as needed
    dataset = make_dataset()  
    print(len(dataset))
    return Task(
        dataset=dataset,
        solver=[multiple_choice()],
        scorer=choice(),
        name=task_name,
    )

#run the following lines from command-line
#export OPENROUTER_API_KEY="your_openrouter_api_key"
#command line eval:
#inspect eval ayubench-trial.py --model openrouter/google/gemini-2.5-flash
#          --model-base-url https://openrouter.ai/api/v1/