Biology Atlas

Map / Concepts

concept · intermediate · partly written

How a link-prediction benchmark is built and how it leaks

A link-prediction benchmark hides true edges and asks a model to rank them back; mirror edges, node degree and random splits inflate the score without any biology learned.

Short answer

Hide some true edges, rank them back, score the rank. Three things inflate the score: test edges whose mirror image is in training, node degree, and random splits. Always ask for a degree-only baseline and a split by time or by disease. 1234

Reported, not yet replicated

The recipe, with real examples more

ogbl-biokg has 93,773 nodes of 5 types, 5,088,434 edges and 51 relation types, with a random split; its page says splitting by time is attractive but dates are unknown. Hetionet's gold standard has 755 positives against 29,044 negatives. TxGNN used three splits: random, zero-shot (random diseases with all their drugs moved to test) and by disease area; its AUPRC was 0.913 versus 0.873 for the best baseline on the random split (authors' own result). 456

Reported, not yet replicated

Leak A: the mirror edge more

In the FB15k general benchmark 70% of test triples had a reverse triple in training, and a one-rule model reached 71.6% FHits@1 against 73.8% for the best embedding model; WN18 figures were 93% and 96.4%. Removing the leak cut one model's FMRR from 0.797 to 0.308. These are general knowledge-graph benchmarks; no leak count for a biomedical graph was found. ogbl-biokg stores symmetric relations as bi-directional edges, which makes the same leak plausible there but unmeasured. 14

Reported, not yet replicated

Leak B: degree more

TransE on Hetionet scored held-out disease-gene answers for high-degree genes (degree above 1000) at MRR 0.053 against 0.006 for low-degree genes (below 200), over 10 splits. A degree-only edge prior reached AUROC 0.7697 to 0.9971 on 20 Hetionet-derived networks, at least 0.95 on 17 of them, but about 0.5 on a separate PPI network. The second study's authors overlap with the Hetionet group and declare industry funding; the first includes AstraZeneca staff. 23

Reported, not yet replicated

Leak C: the split itself more

On a random split a held-out disease still has its drugs in training. Mayers and colleagues found random 5-fold AUROC 0.95 against a time-resolved peak of 0.797 and about 0.70 beyond five years, and advised against validating single candidates. PrimeKG's own time check is a 40-drug spot check, the authors' own assertion. Of 10,938 DrugCentral drug-disease pairs, 5,337 survived identifier mapping in Mayers and colleagues' graph, a loss before any model runs. 78

Reported, not yet replicated

Still open here

Not yet written: Plain answer, Why it matters, Equation, Real numbers.

Linked from

Sources

  1. Akrami F, et al.. Realistic re-evaluation of knowledge graph completion methods: an experimental study. SIGMOD 2020 (arXiv 2003.08001) (2020). paper; opened 2026-11-18 all records citing this
  2. Bonner S, et al.. Implications of topological imbalance for representation learning on biomedical knowledge graphs. arXiv 2112.06567 (2022). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  3. Zietz M, Himmelstein DS, Greene CS, et al.. The probability of edge existence due to node degree: a baseline for network-based predictions. GigaScience (2024). paper; opened 2026-11-18 all records citing this
  4. Open Graph Benchmark team. Open Graph Benchmark: link property prediction datasets (ogbl-biokg). ogb.stanford.edu (2026). web, not peer reviewed; opened 2026-11-18 all records citing this
  5. Himmelstein DS, Lizee A, Hessler C, et al.. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. eLife 6:e26726 (2017). paper; opened 2026-11-18 all records citing this
  6. Huang K, et al., Zitnik M. Zero-shot drug repurposing with geometric deep learning and clinician centered design (TxGNN, medRxiv version 4). medRxiv (PMC11326339) (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  7. Mayers M, Li TS, Queralt-Rosinach N, Su AI. Time-resolved evaluation of compound repositioning predictions on a text-mined knowledge network. BMC Bioinformatics (2019). paper; opened 2026-11-18 all records citing this
  8. Chandak P, Huang K, Zitnik M. Building a knowledge graph to enable precision medicine. Scientific Data 10:67 (2023). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (5)

This record has been changed 5 times, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-19
Record
content/concepts/link-prediction-benchmark-and-leakage.json