Biology Atlas

Map / Concepts

concept · intermediate · partly written

Sakana's AI Scientist-v2 and its experiment tree

The AI Scientist-v2 stores each experiment as a node in a tree and had one of three generated papers accepted at a workshop; an outside test of version 1 found frequent coding failures.

Also called: AI Scientist, Sakana AI Scientist

How it holds knowledge

Each node holds a script, a plan, results and a vision-language-model review. Nodes flagged 'buggy' get debugging children, so a negative result is just a node that is not the best. 1

Reported, not yet replicated

What was shown, and what an outsider found more

One of three generated papers scored 6, 7 and 6 at an ICLR workshop and was withdrawn by pre-agreement; Sakana says none met its main-track bar, and a citation error was admitted. An independent evaluation of version 1 (abstract read) found 42% of experiments failed from coding errors; the version tested is not the v2 described here. 23

Reported, not yet replicated

A baseline that was reported more

Sakana reported trivial baselines for its automated paper reviewer (random and always-reject). Two readings of that figure disagree on the always-reject value (0.50 or 0.59), so neither is used here. 1

Unverified: from our own report, source not opened

Plain answer

The AI Scientist-v2 keeps an experiment tree of code, plan, results and review per node; it is a record of runs, not a store of claims. 1

Reported, not yet replicated

Why it matters

It is the one system here with an outside test that went badly, which shows what builders' own demos leave out. 3

Our inference

Still open here

Not yet written: Equation, Real numbers.

Linked from

Sources

  1. Sakana AI and collaborators. The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv 2504.08066 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  2. The AI Scientist generates its first peer-reviewed workshop paper. Sakana AI (2025). organisation-page, not peer reviewed; opened 2026-11-18 all records citing this
  3. Beel J, Kan MY, Baumgart M. Independent evaluation of Sakana's AI Scientist (version 1). arXiv 2502.14297 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (6)

This record has been changed 6 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-19
Record
content/concepts/ai-scientist-v2-experiment-tree.json