Biology Atlas

Map / Concepts

concept · intermediate · partly written

How agentic science systems hold what they know

A comparison of what Robin, Kosmos, the Google AI co-scientist, the AI Scientist-v2, the Virtual Biotech and Biomni keep between steps, and what they do not keep.

What is stored, system by system

In the papers opened, these systems keep prose and run files, not typed claims with a revision rule. The nearest to typed is the Virtual Biotech, which stores schema-validated JSON with a source tag per annotation. Kosmos says 'structured world model' and publishes no schema. 1234

Reported, not yet replicated

The comparison more

Co-scientist: the unit is a hypothesis plus its reviews, kept in a text context memory, with an Elo rating (new hypotheses start at 1,200); the authors call the rating a proxy, and the Meta-review text is appended to later prompts, not written to a database. Robin: a written report and a ranked hypothesis list in timestamped run folders; ranking is a Bradley-Terry-Luce fit from an LLM judge. Kosmos: summaries of task outputs, labelled by producing agent, form unstated. AI Scientist-v2: a node in an experiment tree (script, plan, results, review), where failed nodes get debugging children. Virtual Biotech: JSON validated against a Pydantic schema. Biomni: no memory; knowledge sits in 150 tools and 59 databases chosen per query. 1563748

Reported, not yet replicated

The empty column: what happens when an experiment contradicts a stored claim more

No paper read states a rule. Read for it: the co-scientist (arXiv and Nature main text), Robin (both versions), Kosmos (full preprint), the AI Scientist-v2 (preprint) and the Virtual Biotech (first ~200,000 characters). Supplements and the Sakana and Science versions were not opened, so this is 'not found in the text read'. Kitano's 2021 paper asked for non-monotonic knowledge and belief revision; no LLM system read does that (the comparison is our reading). 12379

Our inference

Who measured what more

Every wet-lab check found was run by the builders or by collaborators on the author list. Outside tests are small and unflattering: version 1 of Sakana's system had 42% of experiments fail from coding errors in an independent evaluation (abstract read); an outside audit of three Kosmos claims found the DDR-p53 claim not distinguishable from random gene sets (abstract read, n=3, preprint). Independent test of perturbation-prediction foundation models: none beat a 'no change' or additive baseline. 101112

Reported, not yet replicated

Grounding names to identifiers more

Not measured inside any system read. The measured fact is outside: GPT-4o without browsing was wrong on about 99% of Ensembl-ID-to-symbol conversions and about 1% wrong with web access (preprint). Grounding works by lookup. The Virtual Biotech joins by NCT trial IDs, Open Targets IDs and Cell Ontology terms and reports no mapping accuracy. A benchmark audit found FAM138A treated as protein-coding; HGNC lists it as a long non-coding RNA gene. 1341415

Reported, not yet replicated

Plain answer

Between steps these systems keep text: reports, summaries, hypothesis text with an LLM-judged score, or a tree of code runs. None of the papers read gives a revision rule for a claim that a later experiment contradicts, and none reports how accurately it maps gene and drug names to identifiers. 123

Reported, not yet replicated

Why it matters

A discovery system can only be as good as what it can carry from one step to the next. If a negative result, a cell type or an assay is lost in a summary, every later step works without it. 316

Our inference

Still open here

Not yet written: Equation, Real numbers.

Linked from

Sources

  1. Gottweis J, et al.. Towards an AI co-scientist (Nature version). Nature 655:487-496 (2026). paper; opened 2026-11-18 all records citing this
  2. Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery (Nature version). Nature 655:497 (2026). paper; opened 2026-11-18 all records citing this
  3. Edison Scientific and collaborators (37 authors). Kosmos: an AI scientist for autonomous discovery. arXiv 2511.02824 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  4. Zhang, Eckmann, Miao, Mahon, Zou. The Virtual Biotech. bioRxiv (2026). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  5. Gottweis J, Natarajan V, et al.. Towards an AI co-scientist. arXiv 2502.18864 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  6. Ghareeb AE, et al.. Robin: a multi-agent system for automating scientific discovery. arXiv 2505.13400 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  7. Sakana AI and collaborators. The AI Scientist-v2: workshop-level automated scientific discovery via agentic tree search. arXiv 2504.08066 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  8. Huang K, et al.. Biomni: a general-purpose biomedical AI agent. PMC preprint copy (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  9. Kitano H. Nobel Turing Challenge: creating the engine for scientific discovery. npj Systems Biology and Applications (2021). paper; opened 2026-11-18 all records citing this
  10. Beel J, Kan MY, Baumgart M. Independent evaluation of Sakana's AI Scientist (version 1). arXiv 2502.14297 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  11. Nusrat, Nusrat. Null-model audit of three Kosmos discovery claims. arXiv 2511.13825 (2025). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  12. Ahlmann-Eltze C, Huber W, Anders S. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nat Methods (2025). paper; opened 2026-11-17 all records citing this
  13. Hou W, Ji Z. GeneTuring tests GPT models in genomics. PMC / bioRxiv (2023). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  14. Audit of the BixBench Verified-50 tasks. arXiv 2604.24955 (2026). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  15. HGNC REST service (genenames.org): gene symbol lookups. HUGO Gene Nomenclature Committee (2026). database, not peer reviewed; opened 2026-11-17 all records citing this
  16. Skarlinski MD, et al.. Language agents achieve superhuman synthesis of scientific knowledge (PaperQA2). arXiv 2409.13740 (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (6)

This record has been changed 6 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-19
Record
content/concepts/how-agentic-science-systems-hold-knowledge.json