Biology Atlas

Map / Open register

question · intermediate · partly written

Does LLM-judged hypothesis ranking beat a trivial baseline, and has any builder reported one?

Does LLM-judged hypothesis ranking beat a trivial baseline, and has any builder reported one?

What is known

In an independent study the best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50%. Sakana reported random and always-reject baselines for its reviewer. The co-scientist's Elo is not based on independent ground truth in its authors' words. 1234

Reported, not yet replicated

What is not known more

Whether the co-scientist's or Robin's ranking beats a length, citation-count or plain LLM-with-search baseline; no ranking-loop paper read reports one. 1234

Our inference

Observation

Builders report that scores rise or match experts; the only independent ranker test found the best LLM near chance. 1234

Reported, not yet replicated

What is known

In an independent study the best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50%. Sakana reported random and always-reject baselines for its reviewer. The co-scientist's Elo is not based on independent ground truth in its authors' words. 1234

Reported, not yet replicated

What is not

Whether the co-scientist's or Robin's ranking beats a length, citation-count or plain LLM-with-search baseline; no ranking-loop paper read reports one. 1234

Our inference

What would settle it

Run the ranking loop against a plain baseline on goals with known outcomes, and report the order and judge-swap tests. 1234

Our inference

Still open here

Not yet written: Hypotheses, What is missing, Who works on it.

Linked from

Sources

  1. Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  2. Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  3. Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery. arXiv 2505.13400 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  4. Sakana AI and collaborators. The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv 2504.08066 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)

This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Kind
big-open
Register
open-question
Status
partly written
Last updated
2026-11-19
Record
content/questions/agent-hypothesis-ranking-against-trivial-baseline.json