Biology Atlas

Map / Concepts

concept · intermediate · partly written

Science sandbox: a task environment where an agent runs experiments against a hidden oracle

A science sandbox is a packaged world in which an agent submits experiments, a hidden oracle returns reports, and the agent is judged on what it learned.

What does 'sandbox' mean in the Rao and Lander paper?

The paper defines a science sandbox as a closed-loop setting for evaluating how an agent learns through experiment (Methods, 'Definition of a science sandbox'). Results, paragraph 1, calls it a controlled environment in which AI agents can conduct experiments to learn the rules governing a phenomenon. It is a task environment, not a confinement boundary. 1

Reported, not yet replicated

What are the three kinds of oracle? more

A wet oracle is a real experiment. A damp oracle is a model trained on real data. A dry oracle is an invented rule whose answer the builders know. The authors state the costs: wet is costly and noisy, damp inherits the model's biases, and dry lacks real complexity. 1

Reported, not yet replicated

What does the agent do in one round? more

The agent submits a specimen, such as a DNA library or a sequence. A hidden oracle runs an assay on it and returns a short report. The agent revises its hypothesis and tries again. The organisers say the central object of evaluation is the agent's lab notebook, not only the number. 12

Reported, not yet replicated

Is the oracle sealed by a wall? more

No. In the authors' design the scoring pipeline and oracle weights are present in the agent's working environment, and the agent is told not to inspect them. The authors report that their log analysis showed the agents obeyed. The public repository likewise ships the scorer, the weights and the held-out sets in the agent's folder. 13

Reported, not yet replicated

Does the paper use the word for isolation too? deep

Only as plumbing. Agents ran in a sandboxed Linux environment with a shell, Python, a filesystem and internet (Methods, 'Agent configuration'). A separate record covers confinement from hardware upward. 1

Reported, not yet replicated

Who judges the notebooks, and how well? deep

The authors propose independent AI judges to score notebooks against the oracle's true rules, and the organisers' Approach page describes them as proposed. No agreement figure between such a judge and a scientist appears in the text of the paper that was read; the figures and supplementary figures were not viewed. 12

Reported, not yet replicated

Plain answer

In the Rao and Lander paper a science sandbox is a task environment, not an isolation box. The agent submits an experiment, a hidden oracle runs an assay and returns a short report, and the agent revises. The oracle can be a real experiment (wet), a model trained on real data (damp) or an invented rule whose answer the builders hold (dry). 1

Reported, not yet replicated

Why it matters

Anyone who builds or enters a task environment for a science agent has to choose what the agent may see, what the oracle returns and how the result is scored. This record names those choices as the authors made them. 12

Reported, not yet replicated

Real numbers

Hidden evaluation sets in the regulatory-DNA sandbox

The agent's designs are scored on 14 hidden evaluation sets. Five carry real assay labels and nine carry labels generated by the oracle model. 1

Reported, not yet replicated

Library size

The agent designs 50,000 sequences of 200 bases in the one-shot setting. 1

Reported, not yet replicated

Teach this

In one sentence

A science sandbox is a packaged world where an agent runs experiments against a hidden oracle and is judged on whether it learned the rule, not only on the score. 1

Reported, not yet replicated

Still open here

Not yet written: Equation.

Linked from

Sources

  1. Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
  2. BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
  3. science-sandbox (code repository for the sandboxes paper). GitHub (2026). web, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-17
Record
content/concepts/science-sandbox-task-environment.json