The 20% Gap: How AI Benchmarks in Drug Discovery Are Systematically Overstated — And How to Fix It.

An empirical audit of Tox21 MoleculeNet baselines reveals a critical evaluation flaw that affects nearly every published model comparison.

Introduction

Published papers on AI for drug toxicity prediction routinely report ROC-AUC scores above 0.80 on the Tox21 dataset. These numbers look impressive. They should also make you suspicious.

The reason most benchmarks look this good is a quiet methodological choice buried in the experimental setup: random splitting.

This post documents what happens when you switch to the scientifically correct alternative — scaffold splitting — and why the difference matters enormously for the real goal: models that generalize to compounds they have never seen before.

What Is Tox21?

Tox21 is a benchmark dataset containing ~8,000 compounds screened against 12 nuclear receptor and stress response pathways. It is a standard in MoleculeNet — the de facto benchmark suite for molecular machine learning — and it is used in nearly every graph neural network and fingerprint-based toxicity prediction paper.

The 12 tasks include assays like NR-AR (androgen receptor), SR-p53, and SR-MMP. Each compound may have positive, negative, or missing labels per task, creating a multi-task, class-imbalanced problem with significant label sparsity.

The Problem: Random Split vs. Scaffold Split

Random split shuffles molecules uniformly into train/validation/test sets. It is fast, simple, and almost always wrong for molecular ML.

Why? Because similar molecules share the same Murcko scaffold — the core ring system that defines their chemical family. When you randomly split, the train set and test set contain molecules from the same scaffold families. The model memorizes structure-property relationships within a chemical family and appears to “generalize.”

Scaffold split assigns entire scaffold families exclusively to one set. The test set contains scaffold families the model has never encountered during training. This simulates the actual challenge: predicting toxicity for a genuinely novel drug candidate.

The Audit: What I Found

I built a clean pipeline from scratch:

1. Data Sanitization with RDKit Standard DeepChem loaders crash on metal-ion complexes and malformed SMILES in the raw Tox21 CSV. Before any modeling, I ran every SMILES string through RDKit.Chem.MolFromSmiles() and discarded failures. This removed a small but important subset of records that would silently corrupt featurization.

2. ECFP Fingerprints (Radius=2, 1024 bits) Morgan fingerprints capture local atomic neighborhoods. They encode chemical topology without requiring a graph neural network, giving a clean baseline free from architectural choices.

3. MultitaskClassifier (DeepChem-style) A two-layer fully-connected network (1000→500→12) with dropout 0.25, trained with masked binary cross-entropy to handle missing labels correctly.

4. Scaffold Split Implementation Using RDKit.Chem.Scaffolds.MurckoScaffold, I grouped all ~7,800 valid compounds by their core scaffold and assigned scaffold families to train/validation/test (80/10/10) ensuring zero scaffold overlap.

Results:

Split Type__ Mean ROC-AUC (Test)

Random Split____~0.80–0.82

Scaffold Split_____~0.60–0.63

The gap is approximately 0.20 ROC-AUC points. This is not a noise artifact — it reproduces consistently across architectures, random seeds, and fingerprint choices.

The Mistral-7B QLoRA Comparison

As an extension, I evaluated a Mistral-7B model fine-tuned with QLoRA on SMILES-toxicity pairs. The result: under scaffold-split evaluation, the LLM baseline performs comparably to the ECFP + MultitaskDNN baseline.

This has a precise meaning: topology-aware fingerprint models and large language models occupy roughly the same performance tier once you stop inflating scores with random splits.

Published papers that compare GNNs to LLMs using random splits are not measuring generalization. They are measuring interpolation within chemical families.

Why This Matters

If your model will be deployed to screen novel drug candidates — by definition, compounds outside the training distribution — random-split ROC-AUC scores are essentially useless as a reliability metric.

Scaffold-split evaluation is the minimum standard for honest reporting in drug discovery AI. The 0.20 gap this audit exposes is not a small rounding error. It is the difference between a model that generalizes and a model that memorizes.

The Fix: A Reproducible Pipeline

The complete pipeline — sanitization, featurization, splitting, training, evaluation — is available in the linked notebook. It is designed to be:

• Drop-in for MoleculeNet datasets
• Reproducible with fixed random seeds
• Extensible to graph-based featurizers (DeepChem GraphConv, AttentiveFP)

Links

• Notebook: [https://www.kaggle.com/code/sameernadeem66/olmo-smiles-generation-tsm]
• MoleculeNet: moleculenet.org
• DeepChem: deepchem.io

Next: I’ll be extending this audit to BBBP and BACE datasets to check whether the scaffold-split gap holds across other MoleculeNet benchmarks.