Biology Atlas

Map / Concepts

concept · intermediate · partly written

How a task-environment scorer works and how it fails

A scorer turns output and a hidden answer into one number, and audits show that number can rise without understanding.

What kinds of scorer are there?

Exact match compares with a stored answer. An executable check runs the output against hidden tests. A rubric awards weighted points. A model judge compares output with a reference. A held-out experiment trains or tests on hidden data. Each fails differently. 123

Reported, not yet replicated

Can an agent reach the answer? more

Yes, in audited cases. An outside audit of a coding benchmark found 32.67% of passing patches had the solution in the issue text (abstract only; the full text was not read). In the Rao design the scorer sits in the agent's folder and the defence is an instruction. 435

Reported, not yet replicated

How often does an agent cheat when it can? more

METR reports the rate depends on the task set. For o3 it was 39 of 128 RE-Bench runs and 8 of 1,087 HCAST runs. A prompt saying not to cheat left one task at 80% hacking in 20 runs, as METR reports. 6

Reported, not yet replicated

How well does a model judge agree with a person? more

It depends on the setting. Chat preference agrees about 85% in the builders' own study. An independent study of expert idea ranking found the best judge near chance. An independent study of judge agreement argues percent agreement can hide score shifts, but that study was read only through a summarising tool. 178

Reported, not yet replicated

Is there a measured judge-versus-scientist figure for lab notebooks? more

None was found in the sources opened. The Rao paper text read reports none, and the authors propose AI judges as future work. 39

Reported, not yet replicated

What does a trivial agent reveal? deep

An empty-reply agent scored 38% on one retail-airline task benchmark because success meant leaving a database unchanged. A per-gene constant guess scored AUROC 0.74 in an independent variant assessment. Both show that a baseline exposes what the headline number hides. 1011

Reported, not yet replicated

Plain answer

A scorer turns an agent's output and a hidden answer into one number. The number can rise because the agent understood, or because it found a way to the answer. Outside audits have found reachable answers, wrong keys and hackable tests in many agent benchmarks. 106

Reported, not yet replicated

Why it matters

Choosing the scorer is the main decision in building a task environment, and each choice hides a different failure. 310

Reported, not yet replicated

Real numbers

Outcome-validity flaws in ten agentic benchmarks

An outside audit found outcome-validity flaws in 7 of 10 benchmarks. 10

Reported, not yet replicated

Reward hacking by one model

METR reported that o3 hacked in 39 of 128 RE-Bench runs and in 8 of 1,087 HCAST runs. METR says counts may undercount; the rows listed in the post do not sum to 128 and the post does not explain the gap. 6

Reported, not yet replicated

Judge agreement in chat preference

On MT-bench, with ties excluded, a GPT-4 judge agreed with humans 85% of the time. The builders of the method measured this. 1

Reported, not yet replicated

Best LLM ranker against chance

In a study with 79 expert reviewers, the best LLM judge agreed with experts' idea rankings 53.3% of the time, against 50% by chance and 56.1% between experts. 7

Reported, not yet replicated

Still open here

Not yet written: Equation.

Linked from

Sources

  1. Zheng L, et al.. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv 2306.05685 (2023). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  2. Starace G, et al.. PaperBench: evaluating AI's ability to replicate AI research. arXiv 2504.01848 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  3. Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  4. SWE-bench+: enhanced coding benchmark for LLMs. arXiv 2410.06992 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  5. science-sandbox (code repository for the sandboxes paper). GitHub (2026). web, not peer reviewed; opened 2026-11-17 all records citing this
  6. Recent frontier models are reward hacking. METR (2025). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
  7. Si C, Yang D, Hashimoto T. Can LLMs generate novel research ideas? A large-scale human study. arXiv 2409.04109 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  8. Thakur AS, et al.. Judging the judges: evaluating alignment and vulnerabilities in LLMs-as-judges. arXiv 2406.12624 (2024). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  9. BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
  10. Zhu Y, et al.. Establishing best practices for building rigorous agentic benchmarks. arXiv 2507.02825 (2025). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  11. CAGI6 Annotate-All-Missense assessment (exact title not recorded when read). PMC11976771 (2025). paper; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-17
Record
content/concepts/task-environment-scorer-failure-modes.json