ProteinGym: A Unified Benchmark for Protein Fitness Prediction

The Problem with Benchmarking in Computational Biology

Benchmarks are difficult to design well, and in computational biology, the challenge is even greater. Different research groups often rely on their own datasets, evaluation metrics, and experimental setups, making it extremely difficult to compare models in a fair and consistent manner. This fragmentation has been a persistent bottleneck in the field, particularly for protein language models, where rapid advancements have outpaced the development of standardized evaluation frameworks.

ProteinGym was introduced to address this exact issue. It provides a unified, large-scale benchmarking framework where models regardless of architecture or training paradigm are evaluated under identical conditions, on the same datasets, and using consistent metrics. At its core, ProteinGym focuses on protein fitness, capturing how mutations in a protein sequence influence its biological function.

The Core Scientific Challenge

Predicting mutation effects lies at the intersection of three closely connected problems: understanding protein fitness, predicting how mutations alter that fitness, and using those predictions to design better proteins. Protein fitness reflects how well a protein performs its biological role, whether through stability, binding affinity, or catalytic activity. Mutation effect prediction builds on this by determining whether a specific sequence change improves or disrupts function. Protein design extends this further by leveraging these predictions to engineer optimized or entirely novel proteins.

These problems are deeply interconnected. Accurate mutation effect prediction directly enables rational protein design. However, without a standardized benchmark, it has historically been unclear which models are truly reliable. ProteinGym provides that missing foundation.

Two Complementary Benchmarks: DMS and Clinical Data

A key strength of ProteinGym is that it does not rely on a single type of dataset. Instead, it introduces two complementary benchmarking tracks, each capturing a different aspect of protein function.

Deep Mutational Scanning (DMS) Benchmark

The DMS benchmark is based on large-scale experimental assays that systematically measure the effects of mutations on protein function. These datasets contain millions of mutations across diverse proteins, organisms, and functional properties such as stability, binding, and enzymatic activity. Because DMS provides quantitative, high-resolution measurements, it allows models to be evaluated on their ability to capture subtle variations in protein fitness.

Clinical Benchmark

The clinical benchmark is derived from human genetic variation data, primarily curated from databases like ClinVar and gnomAD. These datasets annotate mutations as pathogenic or benign, linking sequence variation directly to disease outcomes. Unlike DMS, which focuses on controlled experimental measurements, clinical data reflects real-world biological consequences, making it essential for evaluating models in a medical context.

Together, these two benchmarks provide a more complete picture: DMS captures experimental fitness landscapes, while clinical data captures real-world disease relevance.

The Three Layers of ProteinGym

ProteinGym is structured around three key layers that together enable standardized evaluation.

The data layer provides the ground truth through DMS assays and clinical annotations, ensuring both experimental and real-world coverage. The model layer includes a wide range of baseline approaches, spanning zero-shot models that rely solely on pre-trained knowledge and supervised models that learn directly from labeled mutation data. The analytics layer defines how performance is measured, using a suite of metrics tailored to different biological and computational scenarios.

This layered design ensures that models are evaluated consistently while still capturing the complexity of protein function.

Mutation Types: Moving Beyond Substitutions

One of ProteinGym’s major contributions is its inclusion of both substitution mutations and indels. Substitutions, where one amino acid is replaced by another, have traditionally been the focus of most benchmarks. However, insertions and deletions introduce changes in sequence length and often lead to more complex structural and functional consequences.

By incorporating indels, ProteinGym significantly increases the realism and difficulty of the benchmarking task, pushing models toward handling biologically relevant scenarios that were previously underexplored.

Zero-Shot vs Supervised Evaluation

ProteinGym evaluates models under two distinct regimes: zero-shot and supervised. In the zero-shot setting, models predict mutation effects without access to any labeled data for the target protein, relying entirely on patterns learned during large-scale pre-training. In the supervised setting, models are trained on a subset of labeled mutations and evaluated on unseen data.

Importantly, all models are tested on mutations they have not seen during training, ensuring a fair comparison. The difference lies only in the information available prior to prediction, not in the evaluation protocol itself.

How Model Performance is Measured

Rather than relying on a single metric, ProteinGym uses a diverse set of evaluation measures tailored to different tasks. For continuous fitness measurements, Spearman correlation assesses how well models capture the relative ordering of mutations by comparing ranks rather than exact values, making it robust to non-linear fitness relationships.

For classification scenarios, AUC and MCC evaluate how effectively models distinguish between functional and non-functional variants. AUC summarizes a model’s ability to separate classes across all thresholds, while MCC provides a more reliable measure in situations with class imbalance, where one class significantly outnumbers the other.

For protein design applications, NDCG@10% and Top-K Recall focus on whether models can correctly identify the highest-performing mutations and rank them at the top of their predictions, which is critical when selecting optimal variants.

Key Insights from ProteinGym

Several important insights emerge from this benchmarking framework. Hybrid models that combine evolutionary information with deep learning approaches tend to perform particularly well. Autoregressive models often outperform masked language models in zero-shot settings, likely due to their ability to compute exact sequence probabilities. Interestingly, zero-shot models can match or even exceed supervised models on clinical datasets, suggesting that supervised approaches may be more prone to overfitting biases in labeled data.

Another critical factor is the availability of evolutionary information. Models perform significantly better when large numbers of homologous sequences are available, highlighting the importance of sequence diversity in learning meaningful biological patterns.

Personal Application

In my Master’s thesis, ProteinGym played a crucial role in model selection. Instead of arbitrarily choosing a protein language model, I used ProteinGym to systematically evaluate multiple candidates under standardized conditions relevant to my specific application. This allowed me to make an informed, evidence-based decision, directly improving the reliability of my downstream analysis.

Limitations and Open Challenges

Despite its strengths, ProteinGym has several limitations. Structure-based models struggle with indels due to fixed-length assumptions, while some language models rely on heuristic methods to estimate mutation effects. Clinical datasets also contain biases, as many annotations are influenced by prior computational predictions. Additionally, modeling interactions between multiple mutations remains a significant challenge, and proteins with limited evolutionary data continue to be difficult to analyze accurately.

A Living Benchmark for the Future

ProteinGym is not static; it continues to evolve as new models and datasets are introduced. Recent updates have expanded the number of evaluated models and highlighted the growing importance of multimodal approaches that integrate sequence, structure, and evolutionary information. These developments suggest that future progress will likely depend more on better integration of biological signals than simply increasing model size.

Conclusion

ProteinGym represents a major step toward standardizing protein fitness prediction. By unifying datasets, evaluation metrics, and benchmarking protocols, it provides a common framework for comparing models across millions of mutations. Whether the goal is to develop new computational methods, understand mutation effects, or design novel proteins, ProteinGym offers a robust and reliable foundation for making scientifically informed decisions.

ProteinGym is comprised of three layers. The data layer encompasses two complementary ground truth labels from DMS assays and clinical annotations from experts. For both, we analyze two types of mutations: substitutions and indels. The model layer is comprised of a diverse set of baselines, tailored to both zero-shot and supervised training regimes. Lastly, the analytics layer includes several performance metrics geared towards fitness prediction or protein design evaluation. Different segmentation variables (e.g., MSA depth, assayed phenotype, taxa) facilitate the comparisons of models across diverse settings

Reference — Notin, P., Kollasch, A. W., Ritter, D., van Niekerk, L., Paul, S., Spinner, H., et al. (2023). ProteinGym: Large-scale benchmarks for protein design and fitness prediction.

Website link of proteinGym — proteingym.org