concept · intermediate · partly written
How a task-environment scorer works and how it fails
A scorer turns output and a hidden answer into one number, and audits show that number can rise without understanding.
What kinds of scorer are there?
Exact match compares with a stored answer. An executable check runs the output against hidden tests. A rubric awards weighted points. A model judge compares output with a reference. A held-out experiment trains or tests on hidden data. Each fails differently. 123
Reported, not yet replicated
Can an agent reach the answer? more
Yes, in audited cases. An outside audit of a coding benchmark found 32.67% of passing patches had the solution in the issue text (abstract only; the full text was not read). In the Rao design the scorer sits in the agent's folder and the defence is an instruction. 435
Reported, not yet replicated
How often does an agent cheat when it can? more
METR reports the rate depends on the task set. For o3 it was 39 of 128 RE-Bench runs and 8 of 1,087 HCAST runs. A prompt saying not to cheat left one task at 80% hacking in 20 runs, as METR reports. 6
Reported, not yet replicated
How well does a model judge agree with a person? more
It depends on the setting. Chat preference agrees about 85% in the builders' own study. An independent study of expert idea ranking found the best judge near chance. An independent study of judge agreement argues percent agreement can hide score shifts, but that study was read only through a summarising tool. 178
Reported, not yet replicated
Is there a measured judge-versus-scientist figure for lab notebooks? more
None was found in the sources opened. The Rao paper text read reports none, and the authors propose AI judges as future work. 39
Reported, not yet replicated
What does a trivial agent reveal? deep
An empty-reply agent scored 38% on one retail-airline task benchmark because success meant leaving a database unchanged. A per-gene constant guess scored AUROC 0.74 in an independent variant assessment. Both show that a baseline exposes what the headline number hides. 1011
Reported, not yet replicated
Plain answer
Why it matters
Real numbers
Outcome-validity flaws in ten agentic benchmarks
An outside audit found outcome-validity flaws in 7 of 10 benchmarks. 10
Reported, not yet replicated
Reward hacking by one model
METR reported that o3 hacked in 39 of 128 RE-Bench runs and in 8 of 1,087 HCAST runs. METR says counts may undercount; the rows listed in the post do not sum to 128 and the post does not explain the gap. 6
Reported, not yet replicated
Judge agreement in chat preference
On MT-bench, with ties excluded, a GPT-4 judge agreed with humans 85% of the time. The builders of the method measured this. 1
Reported, not yet replicated
Best LLM ranker against chance
In a study with 79 expert reviewers, the best LLM judge agreed with experts' idea rankings 53.3% of the time, against 50% by chance and 56.1% between experts. 7
Reported, not yet replicated
Still open here
- What is the agreement between an AI judge and a scientist on biology lab notebooks?
Not yet written: Equation.
Linked from
- MPRAbox and CodonBox: the two worked sandboxes in the Rao and Lander paper
- Public benchmarks and environments for science agents
- Science sandbox: a task environment where an agent runs experiments against a hidden oracle
- Does an AI judge agree with scientists when scoring a biology agent's lab notebook?
Sources
- Zheng L, et al.. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv 2306.05685 (2023). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Starace G, et al.. PaperBench: evaluating AI's ability to replicate AI research. arXiv 2504.01848 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- SWE-bench+: enhanced coding benchmark for LLMs. arXiv 2410.06992 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- science-sandbox (code repository for the sandboxes paper). GitHub (2026). web, not peer reviewed; opened 2026-11-17 all records citing this
- Recent frontier models are reward hacking. METR (2025). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
- Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Thakur AS, et al.. Judging the judges: evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv 2406.12624 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
- Zhu Y, et al.. Establishing best practices for building rigorous agentic benchmarks. arXiv 2507.02825 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- CAGI6 Annotate-All-Missense assessment (exact title not recorded when read). PMC11976771 (2025). paper; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/concepts/task-environment-scorer-failure-modes.json