concept · intermediate · partly written
The Google AI co-scientist and its Elo tournament
Google's AI co-scientist generates hypotheses, ranks them in pairwise LLM debates with an Elo rating, and keeps hypotheses and reviews in a text memory.
Also called: AI co-scientist, Google co-scientist
How it holds knowledge
A persistent text context memory holds hypotheses, reviews, Elo scores, a research plan and feedback appended to prompts. The only graph is a hypothesis-to-hypothesis similarity graph used to schedule tournament matches. Source code is closed, so the real format is unknown. 1
Reported, not yet replicated
How the score is judged more
New hypotheses start at Elo 1,200 and move through pairwise debates judged by LLMs. The paper's own figure caption describes the Elo as auto-evaluated and not based on independent ground truth. The external check is a quiz benchmark (GPQA Diamond, multiple choice): a top-1 accuracy of 78.4% for top-rated answers appears in the arXiv preprint and was not found in the Nature text. Expert ratings (11 goals) come from the experts who wrote the goals. 21
Reported, not yet replicated
What was tested at the bench, and by whom more
Acute myeloid leukaemia drug repurposing: the Nature text lists five candidates chosen by oncologists for wet-lab testing (binimetinib, pacritinib, cerivastatin, pravastatin, dimethyl fumarate) and three autonomous single agents (nanvuranlat, KIRA6, leflunomide). The testing was done by collaborators who are listed authors. Outside academics called the claim vague. No uninvolved group testing the system was found. 13
Reported, not yet replicated
Stated limits deep
The authors say it reads open-access text only, uses figures poorly, did not evaluate multi-omics or knowledge graphs, and miss negative results and paywalled prior art. They call for objective metrics beyond current ranking. 12
Reported, not yet replicated
Plain answer
Why it matters
Real numbers
Independent LLM ranker test
Best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50% (a different study, not the co-scientist). 4
Reported, not yet replicated
Still open here
- Does the Nature supplement report an Elo-versus-expert correlation? Not found in the text read.
- How did the AML candidate figures change between the arXiv and Nature versions?
Not yet written: Equation.
Linked from
Sources
- Gottweis J, et al.. Towards an AI co-scientist (Nature version). Nature 655:487-496 (2026). paper; opened 2026-11-18 all records citing this
- Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Experts don't think AI is ready to be a 'co-scientist'. TechCrunch (2025). news, not peer reviewed; opened 2026-11-18 all records citing this
- Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)
This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-19
- Record
- content/concepts/co-scientist-elo-tournament.json