Biology Atlas

Map / Concepts

concept · intermediate · partly written

The Google AI co-scientist and its Elo tournament

Google's AI co-scientist generates hypotheses, ranks them in pairwise LLM debates with an Elo rating, and keeps hypotheses and reviews in a text memory.

Also called: AI co-scientist, Google co-scientist

How it holds knowledge

A persistent text context memory holds hypotheses, reviews, Elo scores, a research plan and feedback appended to prompts. The only graph is a hypothesis-to-hypothesis similarity graph used to schedule tournament matches. Source code is closed, so the real format is unknown. 1

Reported, not yet replicated

How the score is judged more

New hypotheses start at Elo 1,200 and move through pairwise debates judged by LLMs. The paper's own figure caption describes the Elo as auto-evaluated and not based on independent ground truth. The external check is a quiz benchmark (GPQA Diamond, multiple choice): a top-1 accuracy of 78.4% for top-rated answers appears in the arXiv preprint and was not found in the Nature text. Expert ratings (11 goals) come from the experts who wrote the goals. 21

Reported, not yet replicated

What was tested at the bench, and by whom more

Acute myeloid leukaemia drug repurposing: the Nature text lists five candidates chosen by oncologists for wet-lab testing (binimetinib, pacritinib, cerivastatin, pravastatin, dimethyl fumarate) and three autonomous single agents (nanvuranlat, KIRA6, leflunomide). The testing was done by collaborators who are listed authors. Outside academics called the claim vague. No uninvolved group testing the system was found. 13

Reported, not yet replicated

Stated limits deep

The authors say it reads open-access text only, uses figures poorly, did not evaluate multi-omics or knowledge graphs, and miss negative results and paywalled prior art. They call for objective metrics beyond current ranking. 12

Reported, not yet replicated

Plain answer

The co-scientist keeps hypothesis text and reviews in a context memory and scores hypotheses by Elo in LLM-judged pairwise debates; the authors say the Elo is not based on independent ground truth. 12

Reported, not yet replicated

Why it matters

A rising score from a model judging model output measures agreement with the judge; whether it tracks truth is a separate question. 42

Our inference

Real numbers

Independent LLM ranker test

Best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50% (a different study, not the co-scientist). 4

Reported, not yet replicated

Still open here

Not yet written: Equation.

Linked from

Sources

  1. Gottweis J, et al.. Towards an AI co-scientist (Nature version). Nature 655:487-496 (2026). paper; opened 2026-11-18 all records citing this
  2. Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  3. Experts don't think AI is ready to be a 'co-scientist'. TechCrunch (2025). news, not peer reviewed; opened 2026-11-18 all records citing this
  4. Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)

This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-19
Record
content/concepts/co-scientist-elo-tournament.json