question · intermediate · partly written
Does changing only the data representation change performance, with data and algorithm held fixed?
A test of the claim that algorithms are limited by their representation and representation by their data: this atlas found no clean representation-only comparison.
His claim
"You have innovation at the algorithm layer. I find you can innovate at the algorithm layer all you want, but that can only be as powerful as the data representation. The data representation, of course, can only be as powerful as the data, but at least there's more space to innovate there" 1
Amit's hypothesis
Where changing the data changed the result more
Across GO releases the median agreement of enriched terms with a 2015 reference rose from 0.27 (2004) to 1 (2015) with the algorithm unchanged. Exomiser ranked the diagnosed variant first in 74% of 134 retinal exomes with HPO terms and 3% without, but there the clinician-entered data and the similarity step arrive together, so it is not a representation-only test. 23
Reported, not yet replicated
A case where the evaluation, not the three layers, moved the result more
A clinical entity-linking benchmark fell from 57.47% to 28.06% once training-set overlap was removed, with models and ontology unchanged. A disputed benchmark of rare-disease ranking tools turned on parameter settings: 15% and 26% with default settings against 72% with recommended settings on a different cohort, according to the tool's developers. These suggest a fourth factor, evaluation design. 456
Reported, not yet replicated
A mixed case: algorithm against a sparse representation more
For is-a relations proposed across 10 ontologies, GPT-4 reached recall 0.5 and precision 0.894, against recall 0.337 and precision 1.0 for a reasoner working from the axioms already present. The representation limits what the reasoner can derive, and the language model adds to a sparse one at lower precision. 7
Reported, not yet replicated
What was searched and not found more
A comparison of a flat vocabulary against an ontology on the same patient data with the same ranker was not found. A CAFA-style ablation of the GO hierarchy was not found. Resnik's 1995 work on word taxonomies was left out as not biology. 2
Reported, not yet replicated
Where the evidence stands more
Each layer matters in the settings measured; evaluation design can matter as much as any layer; the ordering the claim proposes, with representation bounded by data and algorithm by representation, is untested by a representation-only comparison. This is a reading of the cases above, and it does not conclude for or against the claim. 234
Our inference
Observation
Hypotheses
Representation caps the algorithm
An algorithm cannot exceed what its representation can express, and the representation cannot exceed its data (his claim, in his words above). 1
Predicts. Swapping only the representation moves performance more than swapping only the algorithm.
Would falsify. A matched comparison in which an algorithm change moves a benchmark further than a representation change on identical data.
Amit's hypothesis
Evaluation design is a further layer
How a benchmark is split and which settings are used can change apparent results as much as any layer. 46
Predicts. Results change when only the split or settings change.
Would falsify. Result stability across splits and settings in matched benchmarks.
Our inference
What is missing
A representation-only test
Matched experiments that vary the representation with data and algorithm fixed. 2
Reported, not yet replicated
What is known
Data and evaluation changes move results; GO release drift is measured. 2
Reported, not yet replicated
What is not
Whether representation alone moves results, and by how much. 3
Reported, not yet replicated
Still open here
- Is there any representation-only comparison for diagnosis ranking (HPO terms against free text, same ranker)?
- Does a CAFA-style ablation of the GO hierarchy exist?
Not yet written: What would settle it, Who works on it.
Linked from
Sources
- Amit's dictation of 2026-11-18: how biology data is represented, and where ontologies can be innovated. Kept in the atlas repository; not published.
- Tomczak A, Mortensen JM, Winnenburg R, Liu C, et al.. Interpretation of biological experiments changes with evolution of the Gene Ontology and its annotations. Scientific reports (2018). paper; opened 2026-11-18 all records citing this
- Cipriani V, Pontikos N, Arno G, et al.. An Improved Phenotype-Driven Tool for Rare Mendelian Variant Prioritization: Benchmarking Exomiser on Real Patient Whole-Exome Data. Genes (Basel) (2020). paper; opened 2026-11-17 all records citing this
- Medical Crossing: a Cross-lingual Evaluation of Clinical Entity Linking.. paper, not peer reviewed; opened 2026-11-18 all records citing this
- Yuan X, Wang J, Dai B, Sun Y, et al.. Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Briefings in bioinformatics (2022). paper; opened 2026-11-18 all records citing this
- Jacobsen JOB, Kelly C, Cipriani V, Robinson PN, et al.. Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Briefings in bioinformatics (2022). paper; opened 2026-11-18 all records citing this
- Toro S, Anagnostopoulos AV, Bello SM, Blumberg K, et al.. Dynamic Retrieval Augmented Generation of Ontologies using Artificial Intelligence (DRAGON-AI). Journal of biomedical semantics (2024). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (6)
This record has been changed 6 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Kind
- amit
- Register
- open-question
- Triage
- needs-research
- Status
- partly written
- Last updated
- 2026-11-19
- Record
- content/questions/does-changing-only-the-representation-change-performance.json