Biology Atlas

Map / Concepts

concept · intermediate · partly written

How a variant benchmark leaks: circularity, gene-level labels and trivial baselines

A variant benchmark leaks when its labels reach the predictor, and measured repairs split by gene, remove training overlap or use an independent assay.

What are the two kinds of circularity?

Type 1 circularity puts the same variants in training and test. Type 2 puts different variants of the same protein on both sides. Grimm and colleagues show type 2 matters because most proteins carry a single class. 1

Reported, not yet replicated

Does a trivial protein-level guess score well? more

Yes. A protein majority vote, which scores a variant by the labels of its protein's other variants, does well without reading the variant. A gene-prior baseline scored AUROC 0.74 in CAGI6. 12

Reported, not yet replicated

What did the leakage-controlled assessment change? more

CAGI6 scored 60 predictors only on variants added after a cut-off date and excluded the same codon. Predictors trained on labelled variants dropped on a gene-balanced subset, and predictors not trained on labels did not. 2

Reported, not yet replicated

How do builders remove the same-variant leak? more

Pejaver and colleagues removed variants found in seven tools' training sets and used only variants absent from an earlier ClinVar file. Livesey and Marsh used deep mutational scans as an independent assay that does not depend on clinical labels, and note that tuning on a test set biases even unsupervised methods. 34

Reported, not yet replicated

How skewed are labels across proteins? deep

ProteinGym reports that half the labels fall in about a tenth of the proteins, so it scores each gene separately and warns that scoring supervised models on all ClinVar labels may overestimate them. 5

Reported, not yet replicated

What would a fair held-out test do? deep

It splits by gene or protein, removes variants in any tool's training data, uses a label date or an independent assay, reports per-class results and publishes a gene-prior baseline. This list follows the measured repairs above; no source states it as a single standard. 1342

Our inference

Plain answer

A variant benchmark leaks when the label, or a near copy, can reach the predictor. Four routes are measured: the same variants in training and test, other variants of the same gene, labels partly made by predictors, and a test set used to choose the model. 143

Reported, not yet replicated

Why it matters

A variant-mechanism environment inherits every one of these routes, plus one more if the agent can look up the answer key. 12

Reported, not yet replicated

Real numbers

Single-class proteins in one benchmark

In Grimm's selected benchmark set, 4,425 of 4,490 proteins (98%) hold only pathogenic or only neutral variants. 1

Reported, not yet replicated

Gene-prior baseline in CAGI6

A baseline that gives each variant its gene's prior pathogenic fraction scored AUROC 0.74 in a 60-predictor assessment of variants added after a cut-off date. 2

Reported, not yet replicated

Still open here

Not yet written: Equation.

Linked from

Sources

  1. Grimm DG, et al.. The evaluation of tools used to predict the impact of missense variants is hindered by two types of circularity. Human Mutation (2015). paper; opened 2026-11-17 all records citing this
  2. CAGI6 Annotate-All-Missense assessment (exact title not recorded when read). PMC11976771 (2025). paper; opened 2026-11-17 all records citing this
  3. Pejaver V, Byrne AB, Feng BJ, et al.. Calibration of computational tools for missense variant pathogenicity classification and ClinGen recommendations for PP3/BP4 criteria. Am J Hum Genet (2022). paper; opened 2026-11-17 all records citing this
  4. Livesey BJ, Marsh JA. Using deep mutational scanning to benchmark variant effect predictors and identify disease mutations. Molecular Systems Biology (2020). paper; opened 2026-11-17 all records citing this
  5. ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. NeurIPS 2023 Datasets and Benchmarks (2023). paper; opened 2026-11-18 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-17
Record
content/concepts/variant-benchmark-circularity-and-leakage.json