Biology Atlas

Map / Open register

question · intermediate · partly written

Does an AI judge agree with scientists when scoring a biology agent's lab notebook?

No measured agreement between an AI judge and a scientist on biology lab notebooks was found.

What is known and what is not?

Judge agreement is measured for chat and for idea ranking, and the best result on expert-level ranking is near chance. For biology lab notebooks no measurement was found. 12

Reported, not yet replicated

Observation

The Rao paper proposes AI judges for lab notebooks, and the organisers describe them as proposed. No measured agreement between such a judge and a scientist was found. 34

Reported, not yet replicated

What is known

The nearest measurements are in other settings: chat preference agrees about 85% in the builders' study, and the best LLM ranker of research ideas agreed with experts 53.3% of the time against 50% by chance. 12

Reported, not yet replicated

What is not

Whether any judge agrees with scientists on biology notebooks was not found in the sources opened. Four PubMed queries returned nothing and a web search was refused, so the absence is weak. 34

Our inference

What would settle it

Score a set of notebooks with scientists and with a judge, report both agreement and the effect on the ranking of agents. 5

Our inference

Still open here

Not yet written: Hypotheses, What is missing, Who works on it.

Linked from

Sources

  1. Zheng L, et al.. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv 2306.05685 (2023). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  2. Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  3. Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  4. BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
  5. Thakur AS, et al.. Judging the judges: evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv 2406.12624 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Kind
big-open
Register
open-question
Status
partly written
Last updated
2026-11-17
Record
content/questions/judge-agreement-with-scientists-on-lab-notebooks.json