concept · intermediate · partly written
How a link-prediction benchmark is built and how it leaks
A link-prediction benchmark hides true edges and asks a model to rank them back; mirror edges, node degree and random splits inflate the score without any biology learned.
Short answer
Hide some true edges, rank them back, score the rank. Three things inflate the score: test edges whose mirror image is in training, node degree, and random splits. Always ask for a degree-only baseline and a split by time or by disease. 1234
Reported, not yet replicated
The recipe, with real examples more
ogbl-biokg has 93,773 nodes of 5 types, 5,088,434 edges and 51 relation types, with a random split; its page says splitting by time is attractive but dates are unknown. Hetionet's gold standard has 755 positives against 29,044 negatives. TxGNN used three splits: random, zero-shot (random diseases with all their drugs moved to test) and by disease area; its AUPRC was 0.913 versus 0.873 for the best baseline on the random split (authors' own result). 456
Reported, not yet replicated
Leak A: the mirror edge more
In the FB15k general benchmark 70% of test triples had a reverse triple in training, and a one-rule model reached 71.6% FHits@1 against 73.8% for the best embedding model; WN18 figures were 93% and 96.4%. Removing the leak cut one model's FMRR from 0.797 to 0.308. These are general knowledge-graph benchmarks; no leak count for a biomedical graph was found. ogbl-biokg stores symmetric relations as bi-directional edges, which makes the same leak plausible there but unmeasured. 14
Reported, not yet replicated
Leak B: degree more
TransE on Hetionet scored held-out disease-gene answers for high-degree genes (degree above 1000) at MRR 0.053 against 0.006 for low-degree genes (below 200), over 10 splits. A degree-only edge prior reached AUROC 0.7697 to 0.9971 on 20 Hetionet-derived networks, at least 0.95 on 17 of them, but about 0.5 on a separate PPI network. The second study's authors overlap with the Hetionet group and declare industry funding; the first includes AstraZeneca staff. 23
Reported, not yet replicated
Leak C: the split itself more
On a random split a held-out disease still has its drugs in training. Mayers and colleagues found random 5-fold AUROC 0.95 against a time-resolved peak of 0.797 and about 0.70 beyond five years, and advised against validating single candidates. PrimeKG's own time check is a 40-drug spot check, the authors' own assertion. Of 10,938 DrugCentral drug-disease pairs, 5,337 survived identifier mapping in Mayers and colleagues' graph, a loss before any model runs. 78
Reported, not yet replicated
Still open here
- Do the symmetric relations in ogbl-biokg leak across splits in the standard loaders? The risk is real by the OGB page but no one has measured it.
- Several claims (a rival model beating TxGNN, KG-Bench, VITAGRAPH, a transductive-baseline claim) were not opened and have been removed.
Not yet written: Plain answer, Why it matters, Equation, Real numbers.
Linked from
- Biomedical knowledge graphs: shapes, edges and what has worked
- Context-dependent edges: storing the conditions under which a statement holds
- Hetionet
- How omics layers are joined: by identifier, sample, cell, position, latent space and graph
- Learned representations against the simple baseline, per omics layer
- PrimeKG
- Reaction-prediction benchmarks: what a headline accuracy measures, and how splits inflate it
- How a variant benchmark leaks: circularity, gene-level labels and trivial baselines
- Does LLM-judged hypothesis ranking beat a trivial baseline, and has any builder reported one?
- What fraction of a graph benchmark score survives a degree-only baseline and a by-disease split?
Sources
- Akrami F, et al.. Realistic re-evaluation of knowledge graph completion methods: an experimental study. SIGMOD 2020 (arXiv 2003.08001) (2020). paper; opened 2026-11-18 all records citing this
- Bonner S, et al.. Implications of topological imbalance for representation learning on biomedical knowledge graphs. arXiv 2112.06567 (2022). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Zietz M, Himmelstein DS, Greene CS, et al.. The probability of edge existence due to node degree: a baseline for network-based predictions. GigaScience (2024). paper; opened 2026-11-18 all records citing this
- Open Graph Benchmark team. Open Graph Benchmark: link property prediction datasets (ogbl-biokg). ogb.stanford.edu (2026). web, not peer reviewed; opened 2026-11-18 all records citing this
- Himmelstein DS, Lizee A, Hessler C, et al.. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. eLife 6:e26726 (2017). paper; opened 2026-11-18 all records citing this
- Huang K, et al., Zitnik M. Zero-shot drug repurposing with geometric deep learning and clinician centered design (TxGNN, medRxiv version 4). medRxiv (PMC11326339) (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
- Mayers M, Li TS, Queralt-Rosinach N, Su AI. Time-resolved evaluation of compound repositioning predictions on a text-mined knowledge network. BMC Bioinformatics (2019). paper; opened 2026-11-18 all records citing this
- Chandak P, Huang K, Zitnik M. Building a knowledge graph to enable precision medicine. Scientific Data 10:67 (2023). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)
This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-19
- Record
- content/concepts/link-prediction-benchmark-and-leakage.json