Self-supervised learning on observational scRNAseq especially with naive reconstruction or masking losses will never deliver a causal model of gene regulation. It will learn statistical structure in expression space, not the mechanisms that generate regulatory change. 1/

If u are building a large model, better make sure it actually reliably trashes the simpler approaches on outcomes that matter or offers unique insights & make sure it's really worth the bang for the buck. Nobody is going to use huge models for marginal or no gains. 6/

People keep confusing predictive representation learning with causal discovery. Without perturbations, temporal information, genetic instruments, anchoring in sequence or strong biologically causal structural assumptions, there will be no causal model of gene regulation. 2/

@elephant_ben @SashaGusevPosts All deep learning models can scale in principle. But the data input & the learning strategy have to match the end goals. It is literally impossible to learn causal models of gene regulation via SSL on observational scRNAseq alone. Adam simply does not get this.

@adamlewisgreen @SashaGusevPosts I'm sorry you think X-Cell is SOTA on metrics that make no sense. And also u literally fine tune the model on perturbation data & test on the easiest prediction task. Your base model is a virtual cell NOT. Come on man.

@SashaGusevPosts We did throw a billion parameters at it, and performance improved monotonically with scale.

It's worth noting that predictive representations (including embeddings from scFMs) can be useful to learn causal models. But without the right data inputs & expt design, it's literally impossible to magically learn biologically & statistically causal models. 3/

@SashaGusevPosts Honestly the fact a simpler model still works suggests to me, there are “laws of cells” and such and brute force alone can’t really get better than that.

Which is pretty cool.

You can always pull out anecdotes from your models representation where some correlation structure reveals causal biology. Eg. Co-expression networks certainly carry nuggets of causal relationships but there are no guarantees. This doesn't make the model causal. 4/

However, one can solve many problems eg cell type annotation, batch correction, perturbation prediction (within specific limits around the training/fine tuning data) without ever learning a causal model. Simple approaches can do extremely well when provided the right data. 5/

On the other hand, if u provide a model that consistently delivers what it promises, everyone will keep their mouths shut & happily use it. Quite easy to make the case. The model should be able to speak for itself. 7/7

@adamlewisgreen it's great to try things, but ridge regression also beat your model (Pearson delta 0.3755 vs 0.41 by Rhaister)

@anshulkundaje There's an issue with people thinking AI models are some kind of magic that obviates the need for proper experimental design.