Teortaxes Details 12,096 Expert Cells in AI Model
Post shares model specs including 42 layers and 288 experts each.
TLDR
@teortaxesTex posted model architecture figures showing 12,096 expert cells across 42 layers with 288 experts per layer. The post lists exact REAP importance scores drawn from 12.59M tokens per layer along with route share and output contribution metrics. It suggests slicing the data by 14 domains that include Russian and vision tasks. The message ends with the question of what can be built using these details and includes an attached image.
Combined views
5.3K
1 Source, first seen 28d ago