Chinese AI Models Scale to 2.8T Parameters via Communication Fixes
Reactions from ranked influencers
2 posts
tapping the sign. k3 design choices are laser-focused on fixing comms.
How come Chinese models jumped from 800-1000B to 2.4-2.8T parameters all of a sudden? Normally when this happens, there's a hardware unlock so they can train & more importantly run these models. Can't imagine they just lacked expertise in training bigger models 6 months ago
this is also the immediate question for EU catch-up: available clusters are struggling with comms/small gpu per nodes/powercap limits. now can we retrodesign models around hardware limitations?