Biology Atlas

Map / Concepts

concept · intermediate · partly written

Learned representations against the simple baseline, per omics layer

Independent tests mostly find single-cell, DNA and protein language models level with or behind a simple baseline, with gains where the data are diverse.

Summary

Two independent tests found single-cell foundation models no better than variable-gene selection or an additive baseline. A test of DNA models found them matching or trailing a one-hot convolutional network, with exceptions. Protein models beat a site-independent baseline modestly, trail alignment-based methods, and clearly win only on diverse sequences. Which model wins also depends on who runs the test. 1234

Reported, not yet replicated

Single-cell foundation models, zero-shot more

Kedzierska and colleagues (Microsoft Research and Oxford affiliated authors, testing other groups' models) ran highly variable gene (HVG) selection, scVI and Harmony as baselines. They report that HVG outperforms Geneformer and scGPT across all metrics, and that scGPT won on one dataset not in its pretraining. 1

Reported, not yet replicated

Perturbation prediction more

Ahlmann-Eltze, Huber and Anders compared scGPT, scFoundation, scBERT, Geneformer, UCE, GEARS and CPA with an additive sum of single effects, a no-change model, the training mean and a linear model on four datasets, all from cancer cell lines. None outperformed the baselines, and embeddings from atlas pretraining gave only a small benefit over random embeddings. 2

Reported, not yet replicated

The ranking depends on the runner more

BioLLM, from a company that builds single-cell tools and so an interested party, found scGPT above PCA and Geneformer, scFoundation and scBERT below PCA, with fine-tuning helping every model. Kedzierska ranks scGPT below HVG zero-shot. The two disagree on protocol (zero-shot versus fine-tuned) and on the runner. 51

Contested

DNA language models more

Tang, Somia, Koo and Yu (a preprint) report that frozen embeddings from pre-trained DNA models do not offer substantial advantages over a one-hot convolutional network. BEND found a mixed picture: histone AUROC 0.76 for the convolutional baseline against 0.78 for the best model, gene finding MCC 0.80 for AUGUSTUS against 0.68, disease variants AUROC 0.56 for DeepSEA against 0.77 for the best model; it is human only and uses frozen models. DART-Eval reports variant-effect AUROC 0.697 for the best fine-tuned model against 0.892 for ChromBPNet. The BEND and DART-Eval figures were not re-opened at the source check. 367

Reported, not yet replicated

Protein language models more

ProteinGym (217 substitution assays, about 2.4 million variants) reports mean Spearman 0.359 for a site-independent model, 0.394 for ESM-1b, 0.407 for the ESM-1v ensemble, 0.414 for ESM-2 650M and 0.401 for ESM-2 15B, 0.439 for EVE and 0.455 for GEMME; size did not help monotonically. The group that built the benchmark also built several models in its table. FLIP found that on mutational landscapes pretraining does not help much when data are plentiful, but on diverse sequences such as thermostability ESM scored 0.65 to 0.78 against 0.15 to 0.24 for ridge regression. PEER found ESM-1b best on 10 of 14 tasks, but its authors also trained the baselines and included no alignment-based one. 489

Reported, not yet replicated

Leakage in protein-interaction prediction more

Bernett, Blumenthal and List found six deep sequence-based interaction predictors scored high on random splits through sequence and node-degree overlap, and on a leakage-free partition performed at random level, best test accuracy 56%. 10

Reported, not yet replicated

Plain answer

Against a simple baseline, learned representations of single cells and of DNA often show no clear gain, and protein language models gain modestly over an alignment-free baseline while trailing alignment-based methods. 134

Reported, not yet replicated

Why it matters

Amit asked how the data representation limits what algorithms can do; these tests measure whether a learned representation adds anything over a trivial one. 2

Our inference

Real numbers

ProteinGym mean Spearman 2026-11-18

Site-independent 0.359; ESM-1b 0.394; ESM-2 650M 0.414; EVE 0.439; GEMME 0.455. 4

Reported, not yet replicated

ESM-1v by alignment depth 2026-11-18

Mean Spearman 0.357 at low depth against 0.473 at high depth. 4

Reported, not yet replicated

Still open here

Not yet written: Equation.

Sources

  1. Kedzierska KZ, Crawford L, Amini AP, Lu AX. Zero-shot evaluation reveals limitations of single-cell foundation models. Genome Biol (2025). paper; opened 2026-11-18 all records citing this
  2. Ahlmann-Eltze C, Huber W, Anders S. Deep-learning-based gene perturbation effect prediction does not yet outperform simple linear baselines. Nat Methods (2025). paper; opened 2026-11-17 all records citing this
  3. Tang Z, Somia N, Yu Y, Koo PK. Evaluating the representational power of pre-trained DNA language models for regulatory genomics. bioRxiv (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  4. ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. NeurIPS 2023 Datasets and Benchmarks (2023). paper; opened 2026-11-18 all records citing this
  5. Qiu P, Chen Q, Qin H, Fang S, et al.. BioLLM: A standardized framework for integrating and benchmarking single-cell foundation models. Patterns (N Y) (2025). paper; opened 2026-11-18 all records citing this
  6. Frederikke Isa Marin, Felix Teufel, Marc Horlacher, Dennis Madsen et al.. BEND: Benchmarking DNA Language Models on biologically meaningful tasks. arXiv (2023). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  7. Aman Patel, Arpita Singhal, Austin Wang, Anusri Pampari et al.. DART-Eval: A Comprehensive DNA Language Model Evaluation Benchmark on Regulatory DNA. arXiv (2024). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  8. FLIP: Benchmark tasks in fitness landscape inference for proteins. CaltechAUTHORS (bioRxiv 2021.11.09.467890) (2021). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  9. Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu et al.. PEER: A Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding. arXiv (2022). preprint, not peer reviewed; opened 2026-11-18 all records citing this
  10. Bernett, J., Blumenthal, D. B., List, M.. Cracking the black box of deep sequence-based protein-protein interaction prediction. Briefings in Bioinformatics (2024). paper; opened 2026-11-17 all records citing this
Where it sits on the map
History of this record (1)

This record has been changed 1 time, on the dates below. The note kept against each change is written for the people who maintain the atlas, in their own shorthand, so it is not shown here; it is in the record itself.

Status
partly written
Last updated
2026-11-18
Record
content/concepts/learned-representations-vs-simple-baselines.json