Prior to release, we shared a version of @cohere North Mini Code with AI engineers and answered some questions.
Here's a quick illustrated walkthrough of the model's architecture and training process.
Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate
Users appreciate Cohere sharing detailed internals of the North Mini Code Model because the transparency helps advance the broader AI community.
The 30 billion parameter mixture of experts model stacks 49 Transformer blocks, the first of which is dense.
The MoE layers have 128 experts, and activate 8 for each token. Leading to 3 billion active parameters.
The self-attention setup interleaves sliding window attention and full attention in 3:1 ratio. Our team describes this choice in "Rope to Nope and Back Again: A New Hybrid Attention Strategy" https://arxiv.org/abs/2501.18795
Prior to release, we shared a version of @cohere North Mini Code with AI engineers and answered some questions.
Here's a quick illustrated walkthrough of the model's architecture and training process.
Small models fill an important niche. They: 1. run on more widely available hardware 2. handle tasks within a certain range of complexity 3. take on sub-tasks that larger models delegate
GGUF quantized version now out kudos to @UnslothAI: https://huggingface.co/unsloth/North-Mini-Code-1.0-GGUF
The 30 billion parameter mixture of experts model stacks 49 Transformer blocks, the first of which is dense.
The MoE layers have 128 experts, and activate 8 for each token. Leading to 3 billion active parameters.
The self-attention setup interleaves sliding window attention and full attention in 3:1 ratio. Our team describes this choice in "Rope to Nope and Back Again: A New Hybrid Attention Strategy" https://arxiv.org/abs/2501.18795
@JayAlammar @cohere Love that you're sharing details on the internals like this. This level of detail is something that helps the whole community advance.