question · intermediate · partly written
Does an AI judge agree with scientists when scoring a biology agent's lab notebook?
No measured agreement between an AI judge and a scientist on biology lab notebooks was found.
What is known and what is not?
Judge agreement is measured for chat and for idea ranking, and the best result on expert-level ranking is near chance. For biology lab notebooks no measurement was found. 12
Reported, not yet replicated
Observation
What is known
What is not
What would settle it
Score a set of notebooks with scientists and with a judge, report both agreement and the effect on the ranking of agents. 5
Our inference
Still open here
- What is the agreement between an AI judge and a scientist on biology lab notebooks?
Not yet written: Hypotheses, What is missing, Who works on it.
Linked from
Sources
- Zheng L, et al.. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv 2306.05685 (2023). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
- Thakur AS, et al.. Judging the judges: evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv 2406.12624 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Kind
- big-open
- Register
- open-question
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/questions/judge-agreement-with-scientists-on-lab-notebooks.json