Biology Atlas

Map / Concepts

concept · intermediate · partly written

Reaction-prediction benchmarks: what a headline accuracy measures, and how splits inflate it

A reaction model is scored by reproducing the one recorded product, and the score falls sharply when the test reactions are made unlike the training ones.

Also called: Molecular Transformer, top-k accuracy in retrosynthesis, Clever Hans in reaction prediction

How much does a reaction model's score fall when the test is made harder?

Molecular Transformer top-1 fell from 90.4% on a random USPTO split to 74.6% when test products were made dissimilar to training. On templates never seen in training, template-free models reached top-10 of only 0.27 to 0.79%. A no-training nearest-precedent lookup scored 37.3% top-1 on USPTO-50k. 123

Reported, not yet replicated

How is the Molecular Transformer trained? more

The Molecular Transformer treats reaction prediction as translation of SMILES text. It has four layers, width 256 and about 12 million parameters, and trained for 48 hours on one P100 GPU. Its top-1 on USPTO was 89.6% after 24 hours, 90.1% after 48 hours and 90.4% with the last 20 checkpoints averaged, and its confidence score flags wrong predictions with an area under the curve of 0.89 per the authors' preprint. The paper was read through a summarising tool. 4

Reported, not yet replicated

How does a random split inflate the score? more

In one study 78% of test products on a random USPTO split lay within Tanimoto similarity 0.5 of a training product. Top-1 for the Molecular Transformer fell from 90.4% to 80.9% when test products had to be below 0.6 similar and to 74.6% below 0.4. A second model, WLDN5, fell from 85.6% to 69.3%. The same study found a model giving the right para product in a Friedel-Crafts case while attributing little to the directing fluorine, which is a right answer for a wrong reason (a Clever Hans case). 1

Reported, not yet replicated

What happens on reactions of a type never seen in training? more

For templates not present in training, top-10 was 0 for the template-based LocalRetro and 0.27 to 0.79% (9 to 26 of 3,277) for template-free models, against 70 to 95% on in-distribution tests. These are different papers and protocols from the similarity-split result above, so only the direction compares. 2

Reported, not yet replicated

What is the trivial baseline, and did it win? more

Retrosim finds the most similar recorded precedent by fingerprint and reuses its template, with no training. It scored 37.3% top-1 and 74.1% top-10 on USPTO-50k with reaction class unknown, and 52.9% top-1 with class given. In the same paper a sequence-to-sequence model scored 37.4% and a template baseline 35.4% with class given, so the lookup beat both. Later top-1 values around 50 to 57% come from papers with differing protocols, so the gap is not a clean measure of progress. 3

Reported, not yet replicated

Plain answer

Top-k accuracy asks whether the one recorded answer appears among a model's k best guesses. It rewards reproducing recorded reactions, so a random split that leaves near-duplicates in the test set makes the score look better than it is. 12

Reported, not yet replicated

Still open here

Not yet written: Why it matters, Equation, Real numbers.

Linked from

Sources

  1. Kovacs et al.. Quantitative interpretation explains machine learning models for chemical reaction prediction and uncovers bias. Nature Communications (2021). paper; opened 2026-11-17 all records citing this
  2. Chen, Jung. arXiv 2403.03960 (Chen and Jung, 2024): retrosynthesis models on templates never seen in training [title not copied from the page]. arXiv (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  3. Connor W. Coley, et al.. Computer-assisted retrosynthesis based on molecular similarity. ACS Central Science (2017). paper; opened 2026-11-17 all records citing this
  4. Philippe Schwaller, et al.. Molecular Transformer: A Model for Uncertainty-Calibrated Chemical Reaction Prediction. ACS Central Science (2019). paper; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-17
Record
content/concepts/reaction-model-benchmarks-and-split-leakage.json