Wondering whether a model’s impressive score will hold up? Machine learning model validation drug discovery work comes down to one test: does it still perform on molecules unlike its training data? Splits, baselines and honest reporting decide that.
| Item | Value |
|---|---|
| Split methods compared in a cancer cell-line study | Four (random, scaffold, Butina, UMAP) |
| Datasets in that study | 60 NCI-60 datasets |
| Molecules per dataset | Approximately 33,000-54,000 |
| Ranking of split difficulty in that study | UMAP hardest, then Butina, then scaffold, then random |
| Coverage bias paper publication date | 09 January 2025 |
These figures come from the UMAP-split study in the Journal of Cheminformatics and the Nature Communications coverage-bias paper. Figures can change as papers are updated, so check the sources before you rely on them.
This guide is one of the spokes of our overview of how ai is used in drug discovery. It is educational only and says nothing about whether any compound works.
Why a high score on a benchmark can mislead
A model is usually scored on held-out molecules. If those molecules closely resemble the training set, the score may overestimate performance on new chemistry.
The Nature Communications paper notes that these models assume no coverage bias, “meaning the data are representative of the true distribution”. When they are not, the authors write, “we may overestimate the power of the resulting machine-learning models for real-world data.”
In practice, a medicinal chemist wants to know about the next series of compounds, not a close analogue of a known one. Validation should be designed around that gap.
How do random, scaffold and cluster splits differ?
How you divide the data into training and test sets can affect the reported result. The common options are:
- Random split: molecules are assigned to train or test by chance, so close analogues often land on both sides.
- Scaffold split: molecules sharing a core structure are kept together, so the test set holds scaffolds the model has not seen.
- Butina split: molecules are clustered by similarity and whole clusters are held out.
- UMAP clustering split: molecules are embedded in a low-dimensional map, clustered, and whole clusters are held out.
The Journal of Cheminformatics study put these side by side. It examined four representative AI models across 60 NCI-60 datasets, each with approximately 33,000-54,000 molecules tested on different cancer cell lines.
Its authors concluded that the UMAP split gave “more challenging and realistic benchmarks for model evaluation, followed by Butina splits, then scaffold splits and closely after random splits.” This is one study on one kind of data, so treat the ranking as a signal, not a law.
Why scaffold splits are not a complete fix
Scaffold splitting is widely used because it sounds rigorous. The cell-line study points out that, “under the scaffold split, molecules with different chemical scaffolds are often similar because such non-identical scaffolds often only differ on a single atom, or one may be a substructure of the other.”
The Nature Communications authors reach a similar view: “even scaffold splits can be misleading with respect to a model’s generalization performance”. They observed that almost all test molecules still had a fairly close neighbour in the training split.
A reader should still ask how similar the test set really is to the training set.
Checks to run beyond the split
A sound validation plan combines several habits. None of them is exotic, but each one is skipped surprisingly often.
- Report similarity between test and training molecules, not just the splitting rule.
- Include a simple baseline, such as a fingerprint model, so you can see how much a complex model adds.
- Repeat with different random seeds and splits, and report the spread.
- Check for duplicates and near-duplicates across the train and test sets.
- Describe the chemical space the training data cover, so the model’s domain of applicability is visible.
The coverage-bias paper adds that the conclusions from a scaffold split “still only apply to the dataset itself and its own (restricted) chemical space.” A model validated on one library says little about another.
Common pitfalls in model validation for drug discovery
Beyond the split itself, several recurring errors can quietly inflate results. Spotting them is a large part of reading a paper critically.
- Leakage through preprocessing: normalising, selecting features or tuning settings on the full dataset before splitting lets test information reach the model.
- Tuning on the test set: trying many model variants and reporting the best score on the same held-out data turns the test set into a training aid.
- Single-run reporting: one split and one seed can look better or worse than the typical result by chance.
- Weak or missing baselines: without a simple comparison, a small gain from a large model is hard to interpret.
These pitfalls may make a model look more capable than it will be on a new project. A paper that names these risks and addresses them directly is generally easier to trust than one that reports a single headline number.
Readers who are not modellers can still ask simple questions. Was the test set chosen before the experiments began, and was it ever used to pick the final model?
Validation depends on the task upstream and downstream
The right test depends on what the model is for. A structure predictor, as explained in how does alphafold work, is judged on different evidence than a property model for screening.
The same holds for choosing biological targets, covered in ai target identification, where retrospective benchmarks can be hard to interpret. Computational validation is also not a substitute for laboratory and clinical testing, and a strong benchmark result does not show that a molecule will be safe or effective.
When you read a claim, look for three things: the data source, the split, and whether prospective experiments, not only retrospective scores, support it.
Prospective validation means making predictions first, then testing them in the laboratory without adjusting the model afterwards.
Nothing here is medical guidance, and treatment decisions should be discussed with your physician or another qualified doctor.
Frequently asked questions
Is a random split ever acceptable?
Show answer
It can serve as a baseline or an easy-case reference. The cell-line study compared random splits with three other methods, but a random split should not be the only evidence of generalisation.
Which split is best?
Show answer
No single split is best for every task. In the cell-line study the UMAP split was the hardest of the four, but the authors’ conclusions apply to their datasets and models.
Does good validation mean a model will find a working drug?
Show answer
No. Validation only shows how well a model predicts the data it was tested on. It says nothing about safety or effectiveness in people.
What is coverage bias?
Show answer
It is a mismatch between the chemical space in the training data and the space where the model will be used. The Nature Communications paper argues it can lead to overestimated real-world performance.
This article is for information only. It is general education about science and the drug industry, and it is not medical, treatment or investment advice.
