concept · intermediate · partly written
Science sandbox: a task environment where an agent runs experiments against a hidden oracle
A science sandbox is a packaged world in which an agent submits experiments, a hidden oracle returns reports, and the agent is judged on what it learned.
What does 'sandbox' mean in the Rao and Lander paper?
The paper defines a science sandbox as a closed-loop setting for evaluating how an agent learns through experiment (Methods, 'Definition of a science sandbox'). Results, paragraph 1, calls it a controlled environment in which AI agents can conduct experiments to learn the rules governing a phenomenon. It is a task environment, not a confinement boundary. 1
Reported, not yet replicated
What are the three kinds of oracle? more
A wet oracle is a real experiment. A damp oracle is a model trained on real data. A dry oracle is an invented rule whose answer the builders know. The authors state the costs: wet is costly and noisy, damp inherits the model's biases, and dry lacks real complexity. 1
Reported, not yet replicated
What does the agent do in one round? more
The agent submits a specimen, such as a DNA library or a sequence. A hidden oracle runs an assay on it and returns a short report. The agent revises its hypothesis and tries again. The organisers say the central object of evaluation is the agent's lab notebook, not only the number. 12
Reported, not yet replicated
Is the oracle sealed by a wall? more
No. In the authors' design the scoring pipeline and oracle weights are present in the agent's working environment, and the agent is told not to inspect them. The authors report that their log analysis showed the agents obeyed. The public repository likewise ships the scorer, the weights and the held-out sets in the agent's folder. 13
Reported, not yet replicated
Does the paper use the word for isolation too? deep
Only as plumbing. Agents ran in a sandboxed Linux environment with a shell, Python, a filesystem and internet (Methods, 'Agent configuration'). A separate record covers confinement from hardware upward. 1
Reported, not yet replicated
Who judges the notebooks, and how well? deep
The authors propose independent AI judges to score notebooks against the oracle's true rules, and the organisers' Approach page describes them as proposed. No agreement figure between such a judge and a scientist appears in the text of the paper that was read; the figures and supplementary figures were not viewed. 12
Reported, not yet replicated
Plain answer
In the Rao and Lander paper a science sandbox is a task environment, not an isolation box. The agent submits an experiment, a hidden oracle runs an assay and returns a short report, and the agent revises. The oracle can be a real experiment (wet), a model trained on real data (damp) or an invented rule whose answer the builders hold (dry). 1
Reported, not yet replicated
Why it matters
Real numbers
Hidden evaluation sets in the regulatory-DNA sandbox
The agent's designs are scored on 14 hidden evaluation sets. Five carry real assay labels and nine carry labels generated by the oracle model. 1
Reported, not yet replicated
Library size
The agent designs 50,000 sequences of 200 bases in the one-shot setting. 1
Reported, not yet replicated
Teach this
In one sentence
A science sandbox is a packaged world where an agent runs experiments against a hidden oracle and is judged on whether it learned the rule, not only on the score. 1
Reported, not yet replicated
Still open here
- Is the oracle sealed in the hackathon's own rounds, or only in the authors' research copy? The pages opened do not say.
Not yet written: Equation.
Linked from
- MPRAbox and CodonBox: the two worked sandboxes in the Rao and Lander paper
- Public benchmarks and environments for science agents
- How a task-environment scorer works and how it fails
- BroadBox science sandboxes and hackathon (Broad Institute)
- Arya S. Rao
- Eric S. Lander
- Does an AI judge agree with scientists when scoring a biology agent's lab notebook?
Sources
- Rao AS, Lander ES, Sabeti PC, et al.. Science sandboxes measure the scientific capability of AI agents. arXiv 2608.30165 (2026). preprint, not peer reviewed; opened 2026-11-17 all records citing this
- BroadBox: approach. sandboxes.broadinstitute.org (2026). organisation-page, not peer reviewed; opened 2026-11-17 all records citing this
- science-sandbox (code repository for the sandboxes paper). GitHub (2026). web, not peer reviewed; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/concepts/science-sandbox-task-environment.json