concept · intermediate · partly written
Reaction-prediction benchmarks: what a headline accuracy measures, and how splits inflate it
A reaction model is scored by reproducing the one recorded product, and the score falls sharply when the test reactions are made unlike the training ones.
Also called: Molecular Transformer, top-k accuracy in retrosynthesis, Clever Hans in reaction prediction
How much does a reaction model's score fall when the test is made harder?
Molecular Transformer top-1 fell from 90.4% on a random USPTO split to 74.6% when test products were made dissimilar to training. On templates never seen in training, template-free models reached top-10 of only 0.27 to 0.79%. A no-training nearest-precedent lookup scored 37.3% top-1 on USPTO-50k. 123
Reported, not yet replicated
How is the Molecular Transformer trained? more
The Molecular Transformer treats reaction prediction as translation of SMILES text. It has four layers, width 256 and about 12 million parameters, and trained for 48 hours on one P100 GPU. Its top-1 on USPTO was 89.6% after 24 hours, 90.1% after 48 hours and 90.4% with the last 20 checkpoints averaged, and its confidence score flags wrong predictions with an area under the curve of 0.89 per the authors' preprint. The paper was read through a summarising tool. 4
Reported, not yet replicated
How does a random split inflate the score? more
In one study 78% of test products on a random USPTO split lay within Tanimoto similarity 0.5 of a training product. Top-1 for the Molecular Transformer fell from 90.4% to 80.9% when test products had to be below 0.6 similar and to 74.6% below 0.4. A second model, WLDN5, fell from 85.6% to 69.3%. The same study found a model giving the right para product in a Friedel-Crafts case while attributing little to the directing fluorine, which is a right answer for a wrong reason (a Clever Hans case). 1
Reported, not yet replicated
What happens on reactions of a type never seen in training? more
For templates not present in training, top-10 was 0 for the template-based LocalRetro and 0.27 to 0.79% (9 to 26 of 3,277) for template-free models, against 70 to 95% on in-distribution tests. These are different papers and protocols from the similarity-split result above, so only the direction compares. 2
Reported, not yet replicated
What is the trivial baseline, and did it win? more
Retrosim finds the most similar recorded precedent by fingerprint and reuses its template, with no training. It scored 37.3% top-1 and 74.1% top-10 on USPTO-50k with reaction class unknown, and 52.9% top-1 with class given. In the same paper a sequence-to-sequence model scored 37.4% and a template baseline 35.4% with class given, so the lookup beat both. Later top-1 values around 50 to 57% come from papers with differing protocols, so the gap is not a clean measure of progress. 3
Reported, not yet replicated
Plain answer
Still open here
- Does a copy-the-nearest-precedent baseline beat published enzymatic models on a sequence-clustered split? Not found for enzymatic data.
- How do these models compare with human chemists' judgement? The relevant chemist test was not opened.
Not yet written: Why it matters, Equation, Real numbers.
Linked from
Sources
- Kovacs et al.. Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias. Nature Communications (2021). paper; opened 2026-11-17 all records citing this
- Chen, Jung. arXiv 2403.03960 (Chen and Jung, 2024): retrosynthesis models on templates never seen in training [title not copied from the page]. arXiv (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Connor W. Coley, et al.. Computer-assisted retrosynthesis based on molecular similarity. ACS Central Science (2017). paper; opened 2026-11-17 all records citing this
- Philippe Schwaller, et al.. Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction. ACS Central Science (2019). paper; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/concepts/reaction-model-benchmarks-and-split-leakage.json