Biology Atlas

Map / Open register

question · intermediate · partly written

Does changing only the data representation change performance, with data and algorithm held fixed?

A test of the claim that algorithms are limited by their representation and representation by their data: this atlas found no clean representation-only comparison.

His claim

"You have innovation at the algorithm layer. I find you can innovate at the algorithm layer all you want, but that can only be as powerful as the data representation. The data representation, of course, can only be as powerful as the data, but at least there's more space to innovate there" 1

Amit's hypothesis

Where changing the data changed the result more

Across GO releases the median agreement of enriched terms with a 2015 reference rose from 0.27 (2004) to 1 (2015) with the algorithm unchanged. Exomiser ranked the diagnosed variant first in 74% of 134 retinal exomes with HPO terms and 3% without, but there the clinician-entered data and the similarity step arrive together, so it is not a representation-only test. 23

Reported, not yet replicated

A case where the evaluation, not the three layers, moved the result more

A clinical entity-linking benchmark fell from 57.47% to 28.06% once training-set overlap was removed, with models and ontology unchanged. A disputed benchmark of rare-disease ranking tools turned on parameter settings: 15% and 26% with default settings against 72% with recommended settings on a different cohort, according to the tool's developers. These suggest a fourth factor, evaluation design. 456

Reported, not yet replicated

A mixed case: algorithm against a sparse representation more

For is-a relations proposed across 10 ontologies, GPT-4 reached recall 0.5 and precision 0.894, against recall 0.337 and precision 1.0 for a reasoner working from the axioms already present. The representation limits what the reasoner can derive, and the language model adds to a sparse one at lower precision. 7

Reported, not yet replicated

What was searched and not found more

A comparison of a flat vocabulary against an ontology on the same patient data with the same ranker was not found. A CAFA-style ablation of the GO hierarchy was not found. Resnik's 1995 work on word taxonomies was left out as not biology. 2

Reported, not yet replicated

Where the evidence stands more

Each layer matters in the settings measured; evaluation design can matter as much as any layer; the ordering the claim proposes, with representation bounded by data and algorithm by representation, is untested by a representation-only comparison. This is a reading of the cases above, and it does not conclude for or against the claim. 234

Our inference

Observation

Measured cases vary data or evaluation; none varies representation alone. 23

Reported, not yet replicated

Hypotheses

Representation caps the algorithm

An algorithm cannot exceed what its representation can express, and the representation cannot exceed its data (his claim, in his words above). 1

Predicts. Swapping only the representation moves performance more than swapping only the algorithm.

Would falsify. A matched comparison in which an algorithm change moves a benchmark further than a representation change on identical data.

Amit's hypothesis

Evaluation design is a further layer

How a benchmark is split and which settings are used can change apparent results as much as any layer. 46

Predicts. Results change when only the split or settings change.

Would falsify. Result stability across splits and settings in matched benchmarks.

Our inference

What is missing

A representation-only test

Matched experiments that vary the representation with data and algorithm fixed. 2

Reported, not yet replicated

What is known

Data and evaluation changes move results; GO release drift is measured. 2

Reported, not yet replicated

What is not

Whether representation alone moves results, and by how much. 3

Reported, not yet replicated

Still open here

Not yet written: What would settle it, Who works on it.

Linked from

Sources

  1. Amit's dictation of 2026-11-18: how biology data is represented, and where ontologies can be innovated. Kept in the atlas repository; not published.
  2. Tomczak A, Mortensen JM, Winnenburg R, Liu C, et al.. Interpretation of biological experiments changes with evolution of the Gene Ontology and its annotations. Scientific reports (2018). paper; opened 2026-11-18 all records citing this
  3. Cipriani V, Pontikos N, Arno G, et al.. An Improved Phenotype-Driven Tool for Rare Mendelian Variant Prioritization: Benchmarking Exomiser on Real Patient Whole-Exome Data. Genes (Basel) (2020). paper; opened 2026-11-17 all records citing this
  4. Medical Crossing: a Cross-lingual Evaluation of Clinical Entity Linking.. paper, not peer reviewed; opened 2026-11-18 all records citing this
  5. Yuan X, Wang J, Dai B, Sun Y, et al.. Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Briefings in bioinformatics (2022). paper; opened 2026-11-18 all records citing this
  6. Jacobsen JOB, Kelly C, Cipriani V, Robinson PN, et al.. Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Briefings in bioinformatics (2022). paper; opened 2026-11-18 all records citing this
  7. Toro S, Anagnostopoulos AV, Bello SM, Blumberg K, et al.. Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI). Journal of biomedical semantics (2024). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (6)

This record has been changed 6 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Kind
amit
Register
open-question
Triage
needs-research
Status
partly written
Last updated
2026-11-19
Record
content/questions/does-changing-only-the-representation-change-performance.json