concept · intermediate · partly written
How a variant benchmark leaks: circularity, gene-level labels and trivial baselines
A variant benchmark leaks when its labels reach the predictor, and measured repairs split by gene, remove training overlap or use an independent assay.
What are the two kinds of circularity?
Type 1 circularity puts the same variants in training and test. Type 2 puts different variants of the same protein on both sides. Grimm and colleagues show type 2 matters because most proteins carry a single class. 1
Reported, not yet replicated
Does a trivial protein-level guess score well? more
Yes. A protein majority vote, which scores a variant by the labels of its protein's other variants, does well without reading the variant. A gene-prior baseline scored AUROC 0.74 in CAGI6. 12
Reported, not yet replicated
What did the leakage-controlled assessment change? more
CAGI6 scored 60 predictors only on variants added after a cut-off date and excluded the same codon. Predictors trained on labelled variants dropped on a gene-balanced subset, and predictors not trained on labels did not. 2
Reported, not yet replicated
How do builders remove the same-variant leak? more
Pejaver and colleagues removed variants found in seven tools' training sets and used only variants absent from an earlier ClinVar file. Livesey and Marsh used deep mutational scans as an independent assay that does not depend on clinical labels, and note that tuning on a test set biases even unsupervised methods. 34
Reported, not yet replicated
How skewed are labels across proteins? deep
ProteinGym reports that half the labels fall in about a tenth of the proteins, so it scores each gene separately and warns that scoring supervised models on all ClinVar labels may overestimate them. 5
Reported, not yet replicated
What would a fair held-out test do? deep
It splits by gene or protein, removes variants in any tool's training data, uses a label date or an independent assay, reports per-class results and publishes a gene-prior baseline. This list follows the measured repairs above; no source states it as a single standard. 1342
Our inference
Plain answer
Why it matters
Real numbers
Single-class proteins in one benchmark
In Grimm's selected benchmark set, 4,425 of 4,490 proteins (98%) hold only pathogenic or only neutral variants. 1
Reported, not yet replicated
Gene-prior baseline in CAGI6
A baseline that gives each variant its gene's prior pathogenic fraction scored AUROC 0.74 in a 60-predictor assessment of variants added after a cut-off date. 2
Reported, not yet replicated
Still open here
- How much does a gene-held-out split change scores on mechanism labels end to end?
Not yet written: Equation.
Linked from
- Public benchmarks and environments for science agents
- How a task-environment scorer works and how it fails
- Variant mechanism labels: loss of function, gain of function and dominant negative
- Can an agent fetch the variant answer key from a public database during a run?
- Is there a public, variant-level, expert-curated set of loss-of-function, gain-of-function and dominant-negative labels?
Sources
- Grimm DG, et al.. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Human Mutation (2015). paper; opened 2026-11-17 all records citing this
- CAGI6 Annotate-All-Missense assessment (exact title not recorded when read). PMC11976771 (2025). paper; opened 2026-11-17 all records citing this
- Pejaver V, Byrne AB, Feng BJ, et al.. Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria. Am J Hum Genet (2022). paper; opened 2026-11-17 all records citing this
- Livesey BJ, Marsh JA. Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations. Molecular Systems Biology (2020). paper; opened 2026-11-17 all records citing this
- ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. NeurIPS 2023 Datasets and Benchmarks (2023). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (1)
This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.
- Status
- partly written
- Last updated
- 2026-11-17
- Record
- content/concepts/variant-benchmark-circularity-and-leakage.json