Users welcome new benchmarks for long-range DNA regulatory AI models predicting disease risk because they see the work as a promising step forward with helpful inductive biases and potential for future iterations.
I am hopeful there will be more details incoming, a chance for others to battle test the models & hopefully this is the beginning of an actual fusion of evolutionary self-supervision with valuable specific specific functional context conditioning. 25/
It is great to see an actual long range regulatory benchmark in the blog post (enhancer-to-gene linking) based on CRISPRi FlowFiSH data. Previous long context DNALMs have avoided these benchmarks cuz those models don't learn reg. elements well or their long range effects 17/
Fine mapping also has its problems but I wud certainly treat MPRA nominations from a cell-line as a stronger gold standard. Not a huge deal cuz the model seems to do overall pretty well (although it is a small set of loci tested). 16/
Also, want to note that from the model description available there are a LOT of inductive biases in the architecture (block conv for motif learning, not a pure Xformer), sparse attention anchoring on functional annotations etc. I am happy to see this. 22/
The AD case study with the MPRA vs fine mapping is a bit weird. The MPRA is treated a bit like a gold standard. MPRAs are often used to "validate" variant nominations. But there is a lot of incoming evidence that MPRAs are not remotely reliable to validate disease variants. 15/
That being said, there are clear conventions for this benchmark i.e. distance stratification & auPR instead of auROC. Will be important to see this & a direct head-to-head against the SOTA (AlphaGenome, rE2G etc) 18/
I think it is foolish to not use freely available biological knowledge, annotations directly or indirectly when designing & training models just to desperately adhere to "bitter lesson". 23/
Once strong models are achieved, the lessons learned can help make them sleeker / faster by replacing special purpose modules with optimal general purpose ones if that really helps. 24/
I wud also love to see future iteration of this model post-trained on massive functional data on the exact Borzoi/AlphaGenome/PromoterAI benchmark suite 19/
But, overall this looks like a promising step forward giving them the benefit of doubt without having seen any of the actual details of the model. 21/
These collectively address difficult benchmarks for non-coding regulatory DNA that are quite orthogonal to ClinVAR and TraitGym but essential to test whether models can effectively predict diverse types of variant effects 20/
Also good to see them focused on biodefense as well. Good luck to the team! 26/26