Lewis Tunstall from Hugging Face introduces Carbon genomic foundation models with Carbon-3B matching top DNA models after 1 trillion token training and over 275 times faster inference
Demo runs on Hugging Face Space alongside the new paper.
#17Thomas Wolf@THOM_WOLFIt turns out DNA modeling is interestingly different from language modeling. Read more in our interactive blogpost/demo and explore our work here
A joint work of the @huggingscience, pre-training and post-training teams here
Leandro von Werra@lvwerraWe are releasing Carbon: a crazy fast DNA model Carbon is 275x faster than the next best model. So fast you can process the whole human genome on a single GPU in <2 days. Here are the tricks we used: When modelling DNA sequences a lot of the performance comes down to tokenizing the sequences in a smart way. BPE tokenizer struggle because there are no whitespaces and character (called base in DNA) level tokenizers waste a lot of compute on too many tokens. Carbon is built with a unique tokenizer: we split sequences in chunks of 6 bases, but during both training and inference we can work with single base resolution. That's similar to having word tokens but resolving them at the character level. All possible thanks to the DNA tokens unique structure. The architecture combined with the tokenizer makes the model 275x faster than the previous SoTA (Evo2) at this size. We built an interactive demo so you can explore how the model can generate DNA sequences, investigate the structure of genes, predict the effect of mutations, generate and fold proteins and even reconstruct parts of the tree of life. https://huggingface.co/spaces/HuggingFaceBio/carbon-demo
Excited to share Carbon, the most efficient foundation models for generative DNA 🧬. Carbon-3B matches the performance of leading DNA models, while being over 275x faster at inference!
We trained Carbon on 1T tokens of high-quality DNA sequences and folded in all the tricks of modern LLMs:
- RMSNorm + SwiGLU + RoPE - long-context expansion - GQA
However, training DNA models is cursed compared to LLMs: most of the public data is noisy, BPE doesn't work, cross-entropy loss blows up after a few hundred billion tokens, and there's basically no public evals for such models :(
We solved all these issues over the past few months and you can read all about them in our interactive explainer: https://huggingface.co/spaces/HuggingFaceBio/carbon-demo
The model is really blazing fast and can even generate the whole human genome (3.1M base pairs) on your laptop
Lewis Tunstall@_lewtunExcited to share Carbon, the most efficient foundation models for generative DNA 🧬. Carbon-3B matches the performance of leading DNA models, while being over 275x faster at inference! We trained Carbon on 1T tokens of high-quality DNA sequences and folded in all the tricks of modern LLMs: - RMSNorm + SwiGLU + RoPE - long-context expansion - GQA However, training DNA models is cursed compared to LLMs: most of the public data is noisy, BPE doesn't work, cross-entropy loss blows up after a few hundred billion tokens, and there's basically no public evals for such models :( We solved all these issues over the past few months and you can read all about them in our interactive explainer: https://huggingface.co/spaces/HuggingFaceBio/carbon-demo
More details can be found in our tech report https://paperswithcode.co/paper/83340
Lewis Tunstall@_lewtunThe model is really blazing fast and can even generate the whole human genome (3.1M base pairs) on your laptop
#1703alex peysakhovich@ALEX_PEYSvery nice, like in many other ai4science examples a great place to improve models is the tokenizer