Biology Atlas

Map / Concepts

concept · intermediate · partly written

Public benchmarks and environments for science agents

Public science-agent benchmarks differ in task, scorer and how far anyone other than the builders has checked them.

Which benchmarks test biology and chemistry agents?

LAB-Bench asks multiple-choice biology questions. BixBench asks open-answer data-analysis questions scored by an LLM judge. ScienceAgentBench scores executed programs. CGBench asks models to assign ClinGen evidence codes from a paper. 1234

Reported, not yet replicated

Which have been measured by someone other than the builders? more

An outside audit of CORE-Bench found 15 task-level errors and 20 tasks answerable from an existing artifact. An outside audit of ten agentic benchmarks found outcome-validity flaws in seven. No outside check was found for LAB-Bench, ScienceAgentBench, MLE-bench or PaperBench in the sources opened. 567

Reported, not yet replicated

What is the weakness of each scorer? more

A multiple-choice key leaves no room for method. An LLM judge's agreement is measured against labels that may come from the builders. A rubric scores only what it lists. A medal depends on the contest's own leaderboard. 891

Our inference

How does a variant benchmark for agents handle the answer key? deep

CGBench gives the model the paper text and no tools. Latch's VariantBench uses a sandbox with software libraries; the text read does not say whether it had internet access. No benchmark opened blocks lookup of ClinVar or ClinGen pages. 410

Reported, not yet replicated

Plain answer

Several public benchmarks measure agents on science tasks. They differ in the task, in how it is scored and in whether anyone other than the builders has measured them. Independent checks exist for only a few. 56

Reported, not yet replicated

Why it matters

A builder of a new task environment can choose a scorer and a split by seeing which published environment made which choice. 6

Reported, not yet replicated

Real numbers

LAB-Bench

Multiple-choice biology questions; the abstract states over 2,400 questions. 1

Reported, not yet replicated

ScienceAgentBench

102 tasks from 44 papers, scored by running the program; the abstract's best score is 32.4% in one version and 42.2% in another. 3

Reported, not yet replicated

MLE-bench

75 Kaggle contests scored by medal; the best reported result is 16.9%. 9

Reported, not yet replicated

PaperBench

20 papers and 8,316 rubric leaves scored by an LLM judge; the best result is 21.0% and the judge scored F1 0.83 against labels made by the authors. 8

Reported, not yet replicated

VariantBench (a company's benchmark)

118 deterministic evaluations; the best pass rate is 42.1% by the company's own report. 10

Reported, not yet replicated

Still open here

Not yet written: Equation.

Linked from

Sources

  1. LAB-Bench: measuring capabilities of language models for biology research. arXiv 2407.10362 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  2. BixBench: a comprehensive benchmark for LLM-based agents in computational biology. arXiv 2503.00096 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  3. ScienceAgentBench: toward rigorous assessment of language agents for data-driven scientific discovery. arXiv 2410.05080 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  4. Queen, Zhang, Zou. CGBench: a benchmark of clinical genetics curation (exact title not recorded when read). arXiv 2510.11985 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  5. Audit of CORE-Bench task errors (exact title not recorded when read). arXiv 2606.26158 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  6. Zhu Y, et al.. Establishing best practices for building rigorous agentic benchmarks. arXiv 2507.02825 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  7. CORE-Bench: fostering the credibility of published research through a computational reproducibility agent benchmark. arXiv 2409.11363 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  8. Starace G, et al.. PaperBench: evaluating AI's ability to replicate AI research. arXiv 2504.01848 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  9. MLE-bench: evaluating machine learning agents on machine learning engineering. arXiv 2410.07095 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  10. VariantBench (Latch). Latch (2026). vendor-page, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-17
Record
content/concepts/science-agent-benchmarks.json