Challenges in Effective Candidate Selection in AI-Based Antibody Design — Our Nipah Virus Experience

Over the last month or so, my group at Silico Biosciences has been participating in the Adaptyv Bio Nipah Binder Competition. We generated 160 scFv sequences that were designed to bind to the Glycoprotein G in Nipah virus.

sbio-nipahgpg-064 in the style of Goodsell, thanks to Adobe Firefly.

I’d like to share some of the challenges in AI-based antibody design and how various evaluation metrics don’t necessarily translate into empirical success in the lab.

Note: This post isn’t a complaint of the Adaptyv Bio’s competition, the way they select which submissions to experimentally evaluate, etc. This is a discussion around the need to develop better computational selection methods as a field.

About Nipah Virus and Glycoprotein G

Nipah virus is a deadly, bat-borne zoonotic virus. It has a high mortality rate (up to 75%) and was first found in Southeast Asia in 1998. Once infected, individuals experience a wide variety of symptoms ranging from fever and headaches to encephalitis and severe respiratory distress.

Glycoprotein G (gpG) on the surface of the virus binds to human ephrin B2 and B3 receptors on human cells, particularly in the nervous system.

In the structure below (PDB: 2VSM), ephrin B2 (in purple) is bound to the viral attachment Glycoprotein G (in grey).

PDB: 2VSM

For the protein binder competition, the goal was simply to develop candidate binders (not necessarily at this ephrin B2 binding site). Then, based on an average ipSAE score, collections will be selected to be experimentally validated through the Adaptyv Bio platform.

What’s ipSAE?

Adaptyv chose to use ipSAE (interaction prediction Score from Aligned Errors) as the metric by which to rank the submitted binders to the competition, reporting it has the “highest predictive power of any single metric for binders”. A meta-analysis of >3k de novo designs found that ipSAE_min (the minimum of the binder-to-target and target-to-binder ipSAE) has an average precision score of 0.37 when discriminating binders versus non-binders. This metric is supposed to outperform previously used metrics like the interface shape complementarity and interface energy.

What We Made

We focused on the creation of heavy and light chain Fv sequences using a variety of tools:

• Our 🦋*peleke-1* suite of antibody language models — To generate many Fv sequences given an annotated antigen sequence (with the interfacing residues with ephrin B2 as the desired epitope).

>Annotated glycoprotein G antigen sequence for peleke-1
MPAENKKVRFENTTSDKGKIPSKVIKSYYGTMDIKKINEGLLDSKILSAFNTVIALLGSIVIIVMNI
MIIQNYTRSTDNQAVIKDALQGIQQQIKGLADKIGTEIGPKVSLIDTSSTITIPANIGLLGSKISQS
TASINENVNEKCKFTLPPLKIHECNISCPNPLPFREYRPQTEGVSNLVGLPNNICLQKTSNQILKPK
LISYTLGQSGTCITDPLLAMDEGYFAYSHLERIG[S][C][S][R]GVSKQRIIGVGEVLDRGDEVP
SLFMTNVWTPPNPNTVYHCSAVYNNEFYYVLCAVSTVGDPI[L]NSTYWSGSLMMTRLAVKPKSNGG
GYNQHQLALRSIEKGRYDKVMPYGPSGIKQGDTLYFPAVGFLVRTEFKYNDSNCPITKC[Q][Y]SK
PENCRLSMG[I][R]PNSHYILRSGLLKYNLSDGENPKVVFIEISDQRLSIGSPSKIYDSLGQPVFY
QAS[F]SWDTMIKFGDVLTVNPLVVNWRNNTVISR[P][G][Q][S][Q]CPRFNTCP[E]IC[W]
[E][G][V]YNDAFLIDRINWISAGVFLDSN[Q][T][A][E]NPVFTVFKDNEILYRAQLASE[D]
T[N][A][Q]KTITNCFLLKNKIWCISLV[E][I][Y]D[T]GDNV[I]RPKLFAVKIPEQCT

• peleke-1 + *ProteinMPNN (or [AntiBMPNN](https://github.com/zeysun/AntiBMPNN)*)— To derive more varied CDR loop sequences for candidates that were predicted to bind at the desired location.

Example derived Fv structures (heavy:green, light:yellow) with mutated CDR loop residues shown in red.

In the above example, the original sbio-nipahgpg-055 candidate scored decently high ipSAE-wise (0.63) and thus its CDR loops were regenerated using AntiBMPNN on the predicted Fv-gpG complex structure from Boltz-2. The resulted in the ipSAE score increasing from 0.63 → 0.69 → 0.71, respectively.

Our Best Fv Candidates

We created 160 Fv candidates and evaluated them using the methods listed below. Our top 10 (in terms of binding affinity and on-target binding) are:

Top 10 candidates by best (lowest) HADDOCK3 score, filtered to those at the desire epitope.

All of the above are predicted to bind to the binding site of ephrin B2, perhaps exhibiting desired inhibitory properties. (These all have strong binding affinity metrics: overall binding score <-67; Van der Waals energy <-23 kcal/mol; buried surface area >2,000Å; etc.)

How Do We Know What Good Looks Like?

Empirical validation of in silico-based protein design remains the most expensive part of the drug discovery process. So, while we can generate thousands of candidates in a few minutes for little money, the ability to test them all in the lab would require extensive and expensive resources and time. Thus, we must be picky as to what we send to the lab.

Whether you’re developing tiny binders or full antibodies, there is a great need to appropriately rank candidates for lab-based testing. Today, it’s challenging to computationally filter out bad candidates or promote good candidates as the ranking might vary depending on what metric you use (explained below).

For all of our generated scFV candidates, we used Boltz-2 (via my Docker image) to predict the structure complexes with the antibody bound to the gpG antigen.

Problem 1: ipSAE Scores ≠ ipSAE Scores

The Adaptyv team provided a GitHub repo that outlines their ipSAE scoring logic. So, we used that logic to locally score all of our candidates before submitting to Proteinbase (an attempt to predict how they’ll score in the competition). This proved challenging…

Not only do the ipSAE scores not match — they’re not even correlated.

Proteinbase ipSAE vs. Locally-Computed ipSAE

This made it very difficult to know how our submissions were going to score on the Proteinbase competition site given that we couldn’t replicate the scoring locally.

Problem 2: Binding Affinity ≠ ipSAE Scores

For all of the predicted complexes, we used HADDOCK3's scoring function (as described here) to generate physics-based binding affinity predictions. This includes metrics like electrostatic energy, buried surface area, and Van er Waals energy along with composite scores.

ipSAE Scores vs. Predicted Binding Affinity Metrics

As you can see, there’s no correlation between ipSAE and any of the HADDOCK-based binding affinity metrics.

This was surprising given that we’ve used HADDOCK for plenty of other studies and showed reliable translation of these binding affinity predictions versus wet lab (or real world) antibody performance.

🤷🏼‍♂️ So, what gives?

Potential Culprit: Boltz-2 Structure Prediction Variations

Before the release of AI-based folding models that could generate protein-protein complexes (like AlphaFold2-multimer, AlphaFold3, and Boltz-2), we previously used physics-based simulations to dock to proteins together. While this was slower and more computationally intense, the predicted docking was often very accurate. Moreover, if you run physics-based experiments over and over again, the results are often quite consistent, especially for true binders.

Now, with multimer folding models, we can get results in a few seconds to minutes, but the variation in predictions may pose a problem. We know that predicted structures are just snapshots of a variety of conformations or binding positions, but such inconsistency drastically affects binding metrics.

Let’s take a look at one of our candidates: sbio-nipahgpg-028. The locally predicted Boltz-2 structure places the antibody at the desired epitope (at the same location as ephrin B2) and results in a similar ipSAE score. However, the Boltz-2 structure generated on Proteinbase places the antibody at a totally different site (bound to an undesired domain) and results in a much lower ipSAE score.

But wait, sometimes the Boltz-2 predictions can be quite similar between folding jobs and we still get drastically different ipSAE scores. In the example below, sbio-nipahgpg-114 was a really good candidate according to Proteinbase, but scored horribly in my local folding.

Also, the positioning of portions of the protein by Boltz-2 drastically changed the binding position of our antibodies (and presumably other participants’ binders), affecting ipSAE and all other binding metrics. Note the example below where Boltz-2 has placed the N-terminal “stalk domain” in two different conformations.

Example predictions of the Glycoprotein G from Boltz-2. (Note: these are both aligned to PDB 2VSM chain A, colored by pLDDT).

We saw instances of high ipSAE (and high binding affinity) scores when our antibody was predicted to bind to the ephrin B2 epitope when the stalk was pointed upward, though this shouldn’t happen in vivo when the glycoprotein is sticking outward from the membrane of a viral particle.

How Can We Fix This?

Aside from the obvious room for improvement in folding models, what can we do to improve the scoring of computationally designed protein binders?

1. We should score co-folded structures numerous times to get a distribution of binding performance based on various folding predictions. (i.e., “Average q metric over k folding trials.”)
2. We need more protein-protein interaction data that is paired with experimental structures to develop better prediction methods.

The datasets below constitute <200k records of various binders, but they all lack experimental structures and instead rely on folded structures:

• From the ipSAE preprint, the “experimentally characterised binders” dataset: https://zenodo.org/records/15722219
• A-AlphaBio‘s newly released protein-protein interaction dataset on Hugging Face: https://huggingface.co/datasets/aalphabio/open-alphaseq
• Proteinbase’s downloadable dataset of 1,000+ experimentally validated binders: https://proteinbase.com/download

Thus, if we’re ever going to create more accurate methods to predict an AI-designed protein’s likelihood of binding to a target with sufficient affinity, we need experimentally validated structures (both binder and target) and affinity metrics (Kd, etc.)

Alternatively, we can revert back to relying on molecular dynamics and physics-based simulations, but these come at a computational cost and are much slower.

If you’re interested in exploring the creation of new methods for predicting AI binder success, reach out! ✉ colby@tuple.xyz

Lastly, I’d like to applaud Adaptyv Bio on their community science approach with these competitions and in facilitating the discovery of inhibitory binders for Nipah virus along with the 600+ teams that contributed to this effort. It’s been really amazing to see all of the different protein design methods and models (which I’m dying to try out). 😅

Competitions like these highlight the need for consistent benchmarks and metrics so that our AI-based designs can be better evaluated at scale. This also highlights how far the science has come and I’m excited to see where we go in 2026!

design-fold-bind-repeat

If you’ve read this far and would like to vote for some of our candidates to be experimentally validated, click here: https://proteinbase.com/collections/nipah-binder-competition-submission-7-2Ksfm_7s3M

Simply create an account and click the (⭐ Starred) button on our collection.

Stay curious…