question · intermediate · partly written
Does LLM-judged hypothesis ranking beat a trivial baseline, and has any builder reported one?
Does LLM-judged hypothesis ranking beat a trivial baseline, and has any builder reported one?
What is known
In an independent study the best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50%. Sakana reported random and always-reject baselines for its reviewer. The co-scientist's Elo is not based on independent ground truth in its authors' words. 1234
Reported, not yet replicated
What is not known more
Whether the co-scientist's or Robin's ranking beats a length, citation-count or plain LLM-with-search baseline; no ranking-loop paper read reports one. 1234
Our inference
Observation
What is known
In an independent study the best LLM pairwise ranker agreed with human scores 53.3% of the time, humans with each other 56.1%, random 50%. Sakana reported random and always-reject baselines for its reviewer. The co-scientist's Elo is not based on independent ground truth in its authors' words. 1234
Reported, not yet replicated
What is not
What would settle it
Still open here
- What is the Elo or Bradley-Terry score of a trivial baseline on the same goals?
Not yet written: Hypotheses, What is missing, Who works on it.
Linked from
Sources
- Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery. arXiv 2505.13400 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Sakana AI and collaborators. The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv 2504.08066 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)
This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Kind
- big-open
- Register
- open-question
- Status
- partly written
- Last updated
- 2026-11-19
- Record
- content/questions/agent-hypothesis-ranking-against-trivial-baseline.json