concept · intermediate · partly written
Public benchmarks and environments for science agents
Public science-agent benchmarks differ in task, scorer and how far anyone other than the builders has checked them.
Which benchmarks test biology and chemistry agents?
LAB-Bench asks multiple-choice biology questions. BixBench asks open-answer data-analysis questions scored by an LLM judge. ScienceAgentBench scores executed programs. CGBench asks models to assign ClinGen evidence codes from a paper. 1234
Reported, not yet replicated
Which have been measured by someone other than the builders? more
An outside audit of CORE-Bench found 15 task-level errors and 20 tasks answerable from an existing artifact. An outside audit of ten agentic benchmarks found outcome-validity flaws in seven. No outside check was found for LAB-Bench, ScienceAgentBench, MLE-bench or PaperBench in the sources opened. 567
Reported, not yet replicated
What is the weakness of each scorer? more
A multiple-choice key leaves no room for method. An LLM judge's agreement is measured against labels that may come from the builders. A rubric scores only what it lists. A medal depends on the contest's own leaderboard. 891
Our inference
How does a variant benchmark for agents handle the answer key? deep
CGBench gives the model the paper text and no tools. Latch's VariantBench uses a sandbox with software libraries; the text read does not say whether it had internet access. No benchmark opened blocks lookup of ClinVar or ClinGen pages. 410
Reported, not yet replicated
Plain answer
Why it matters
A builder of a new task environment can choose a scorer and a split by seeing which published environment made which choice. 6
Reported, not yet replicated
Real numbers
LAB-Bench
Multiple-choice biology questions; the abstract states over 2,400 questions. 1
Reported, not yet replicated
ScienceAgentBench
102 tasks from 44 papers, scored by running the program; the abstract's best score is 32.4% in one version and 42.2% in another. 3
Reported, not yet replicated
MLE-bench
75 Kaggle contests scored by medal; the best reported result is 16.9%. 9
Reported, not yet replicated
PaperBench
20 papers and 8,316 rubric leaves scored by an LLM judge; the best result is 21.0% and the judge scored F1 0.83 against labels made by the authors. 8
Reported, not yet replicated
VariantBench (a company's benchmark)
118 deterministic evaluations; the best pass rate is 42.1% by the company's own report. 10
Reported, not yet replicated
Still open here
- Which public benchmarks for science agents have been re-scored by a group that did not build them?
Not yet written: Equation.
Linked from
Sources
- LAB-Bench: measuring capabilities of language models for biology research. arXiv 2407.10362 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- BixBench: a comprehensive benchmark for LLM-based agents in computational biology. arXiv 2503.00096 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. arXiv 2410.05080 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Queen, Zhang, Zou. CGBench: a benchmark of clinical genetics curation (exact title not recorded when read). arXiv 2510.11985 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Audit of CORE-Bench task errors (exact title not recorded when read). arXiv 2606.26158 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Zhu Y, et al.. Establishing best practices for building rigorous agentic benchmarks. arXiv 2507.02825 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- CORE-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv 2409.11363 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Starace G, et al.. PaperBench: evaluating AI's ability to replicate AI research. arXiv 2504.01848 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- MLE-bench: evaluating machine learning agents on machine learning engineering. arXiv 2410.07095 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- VariantBench (Latch). Latch (2026). vendor-page, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/concepts/science-agent-benchmarks.json