concept · intermediate · partly written
How agentic science systems hold what they know
A comparison of what Robin, Kosmos, the Google AI co-scientist, the AI Scientist-v2, the Virtual Biotech and Biomni keep between steps, and what they do not keep.
What is stored, system by system
In the papers opened, these systems keep prose and run files, not typed claims with a revision rule. The nearest to typed is the Virtual Biotech, which stores schema-validated JSON with a source tag per annotation. Kosmos says 'structured world model' and publishes no schema. 1234
Reported, not yet replicated
The comparison more
Co-scientist: the unit is a hypothesis plus its reviews, kept in a text context memory, with an Elo rating (new hypotheses start at 1,200); the authors call the rating a proxy, and the Meta-review text is appended to later prompts, not written to a database. Robin: a written report and a ranked hypothesis list in timestamped run folders; ranking is a Bradley-Terry-Luce fit from an LLM judge. Kosmos: summaries of task outputs, labelled by producing agent, form unstated. AI Scientist-v2: a node in an experiment tree (script, plan, results, review), where failed nodes get debugging children. Virtual Biotech: JSON validated against a Pydantic schema. Biomni: no memory; knowledge sits in 150 tools and 59 databases chosen per query. 1563748
Reported, not yet replicated
The empty column: what happens when an experiment contradicts a stored claim more
No paper read states a rule. Read for it: the co-scientist (arXiv and Nature main text), Robin (both versions), Kosmos (full preprint), the AI Scientist-v2 (preprint) and the Virtual Biotech (first ~200,000 characters). Supplements and the Sakana and Science versions were not opened, so this is 'not found in the text read'. Kitano's 2021 paper asked for non-monotonic knowledge and belief revision; no LLM system read does that (the comparison is our reading). 12379
Our inference
Who measured what more
Every wet-lab check found was run by the builders or by collaborators on the author list. Outside tests are small and unflattering: version 1 of Sakana's system had 42% of experiments fail from coding errors in an independent evaluation (abstract read); an outside audit of three Kosmos claims found the DDR-p53 claim not distinguishable from random gene sets (abstract read, n=3, preprint). Independent test of perturbation-prediction foundation models: none beat a 'no change' or additive baseline. 101112
Reported, not yet replicated
Grounding names to identifiers more
Not measured inside any system read. The measured fact is outside: GPT-4o without browsing was wrong on about 99% of Ensembl-ID-to-symbol conversions and about 1% wrong with web access (preprint). Grounding works by lookup. The Virtual Biotech joins by NCT trial IDs, Open Targets IDs and Cell Ontology terms and reports no mapping accuracy. A benchmark audit found FAM138A treated as protein-coding; HGNC lists it as a long non-coding RNA gene. 1341415
Reported, not yet replicated
Plain answer
Between steps these systems keep text: reports, summaries, hypothesis text with an LLM-judged score, or a tree of code runs. None of the papers read gives a revision rule for a claim that a later experiment contradicts, and none reports how accurately it maps gene and drug names to identifiers. 123
Reported, not yet replicated
Why it matters
Still open here
- Does any of these systems have a revision rule in its supplements or code that the main texts do not show?
- What does the Kosmos world model hold? The paper and company pages do not say.
Not yet written: Equation, Real numbers.
Linked from
- Sakana's AI Scientist-v2 and its experiment tree
- The Google AI co-scientist and its Elo tournament
- Kosmos and its 'structured world model'
- How PaperQA2 turns a paper into something an agent reasons over
- Robin, FutureHouse's multi-agent discovery system
- The Virtual Biotech
- What happens to a stored claim in an agent system when a later experiment contradicts it?
- Where does a negative result live in an agent system's memory?
Sources
- Gottweis J, et al.. Towards an AI co-scientist (Nature version). Nature 655:487-496 (2026). paper; opened 2026-11-18 all records citing this
- Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery (Nature version). Nature 655:497 (2026). paper; opened 2026-11-18 all records citing this
- Edison Scientific and collaborators (37 authors). Kosmos: an AI scientist for autonomous discovery. arXiv 2511.02824 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Zhang, Eckmann, Miao, Mahon, Zou. The Virtual Biotech. bioRxiv (2026). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery. arXiv 2505.13400 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Sakana AI and collaborators. The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv 2504.08066 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Huang K, et al.. Biomni: a general-purpose biomedical AI agent. PMC preprint copy (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Kitano H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Systems Biology and Applications (2021). paper; opened 2026-11-18 all records citing this
- Beel J, Kan MY, Baumgart M. Independent evaluation of Sakana's AI Scientist (version 1). arXiv 2502.14297 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Nusrat, Nusrat. Null-model audit of three Kosmos discovery claims. arXiv 2511.13825 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Ahlmann-Eltze C, Huber W, Anders S. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nat Methods (2025). paper; opened 2026-11-17 all records citing this
- Hou W, Ji Z. GeneTuring tests GPT models in genomics. PMC / bioRxiv (2023). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Audit of the BixBench Verified-50 tasks. arXiv 2604.24955 (2026). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- HGNC REST service (genenames.org): gene symbol lookups. HUGO Gene Nomenclature Committee (2026). database, not peer reviewed; opened 2026-11-17 all records citing this
- Skarlinski MD, et al.. Language agents achieve superhuman synthesis of scientific knowledge (PaperQA2). arXiv 2409.13740 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (6)
This record has been changed 6 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-19
- Record
- content/concepts/how-agentic-science-systems-hold-knowledge.json