Lucas Beyer’s anonymous poll surfaces a real tension inside AI labs: Mixture of Experts models keep delivering stronger results at lower active compute, yet plenty of researchers still find the routing machinery visually and conceptually off-putting.
Wow the minds are actually pretty evenly split! Especially if you consider the last two options to be almost the same, we're at 30/40/25 ish.
If you wonder about myself, I think "yuck" but unfortunately in the current day and age, they are too good to ignore. Hope it'll change.
I wonder what people really think about MoEs? It's ok, the voting is anonymous, you can select the option that you really think, deep inside your heart.
Out of 2,588 votes, roughly one-third called MoEs elegant while 41.5 percent openly disliked them and wished dense models performed better instead.
Practitioners described routing and load-balancing code as frustrating hacks, even while acknowledging the efficiency gains that come from activating only a subset of experts per token.
Many users dismissed Mixture of Experts models as non-differentiable monsters that are too handcoded and unreliable to debug in agent systems.
No Digg Deeper questions have been answered for this story yet.
Off the top of the dome, I have two qualms about them:
1) on a high level, for language, not sure I like the "memorize the whole internet" anymore, ideally we make "learn to reason and use search/knowledge-base" work well enough. I don't even think this is controversial anymore nowadays, though it was my opinion for a while now.
2) even ignoring the former; routing, load-balancing, and the issue when batching (people always forget in reality inference is never a single sequence alone!), all point to it really not being the right form-factor for modern hardware to achieve the "subset active" optimally.
That being said I'm always a "pragmatic idealist" so I'm totally fine with them currently as they are the best bang-for-buck, but I am at the same time waiting/striving for the (imo) more ideal arch.
@giffmana Why yuck? There is something elegant in having only a subset of parameters activated per request no?
@giffmana Why yuck? There is something elegant in having only a subset of parameters activated per request no?
Wow the minds are actually pretty evenly split! Especially if you consider the last two options to be almost the same, we're at 30/40/25 ish.
If you wonder about myself, I think "yuck" but unfortunately in the current day and age, they are too good to ignore. Hope it'll change.
@giffmana for 1) btw mixture of depth kinda make more sense
@giffmana For 2) you are saying this bc in prod some expert are much more frequent than others?
For 1) I actually thought about this the other day (role of MLP when knowledge is useless bc search) i don’t think anyone have tried to quantify that?? Would be super interesting
@giffmana what would you prefer? Dense models are ultimately too either too small and thus clueless, or uneconomical vs MoE, and scaling failed to change this. I had hopes for natural ReLU sparsification (doesn't solve training costs, but could help with inference)
Wow the minds are actually pretty evenly split! Especially if you consider the last two options to be almost the same, we're at 30/40/25 ish.
If you wonder about myself, I think "yuck" but unfortunately in the current day and age, they are too good to ignore. Hope it'll change.
@teortaxesTex race-condition, see answer here :)
Off the top of the dome, I have two qualms about them:
1) on a high level, for language, not sure I like the "memorize the whole internet" anymore, ideally we make "learn to reason and use search/knowledge-base" work well enough. I don't even think this is controversial anymore nowadays, though it was my opinion for a while now.
2) even ignoring the former; routing, load-balancing, and the issue when batching (people always forget in reality inference is never a single sequence alone!), all point to it really not being the right form-factor for modern hardware to achieve the "subset active" optimally.
That being said I'm always a "pragmatic idealist" so I'm totally fine with them currently as they are the best bang-for-buck, but I am at the same time waiting/striving for the (imo) more ideal arch.
@t_cmtl @eliebakouch @giffmana exactly
@giffmana For 2) you are saying this bc in prod some expert are much more frequent than others?
For 1) I actually thought about this the other day (role of MLP when knowledge is useless bc search) i don’t think anyone have tried to quantify that?? Would be super interesting
Off the top of the dome, I have two qualms about them:
1) on a high level, for language, not sure I like the "memorize the whole internet" anymore, ideally we make "learn to reason and use search/knowledge-base" work well enough. I don't even think this is controversial anymore nowadays, though it was my opinion for a while now.
2) even ignoring the former; routing, load-balancing, and the issue when batching (people always forget in reality inference is never a single sequence alone!), all point to it really not being the right form-factor for modern hardware to achieve the "subset active" optimally.
That being said I'm always a "pragmatic idealist" so I'm totally fine with them currently as they are the best bang-for-buck, but I am at the same time waiting/striving for the (imo) more ideal arch.
@giffmana @eliebakouch retrieval transformer didn't pan out :( and not for the lack of trying
@giffmana (mixture of depth is adaptive compute tho not sparsity so v different btw)
@giffmana for 1) btw mixture of depth kinda make more sense
@eliebakouch @giffmana it's too handcoded
@giffmana Why yuck? There is something elegant in having only a subset of parameters activated per request no?
@t_cmtl @eliebakouch @giffmana this guy is pilled
@yacineMTB @t_cmtl @giffmana so you do pruning per domain after? Not sure I get it, the router is a learned parameter (with also constraint of LB but still)
@t_cmtl @eliebakouch @giffmana exactly
@giffmana @eliebakouch Especially (2) is very hard to get correct, and it has many implications when you build a single- or multi-agent system and deploy it in production, particularly when reliability is important. The stochasticity due to (2) makes it really hard to debug issues.
@giffmana it's honestly fascinating to see how people feel about this.
@giffmana @eliebakouch similarly phi models
@giffmana 笑死，AI已经变成"恶心但好用"了，跟速食爱情差不多。
@giffmana For technical/logical tasks MoE (programming, math) For creativity, dense. (Finance, law)
I think all big labs serve MoE, Ant’s is denser than OAI, reflecting on the above ..
@giffmana @teortaxesTex then you need very sparse attention (or whatever equivalent mechanism for reasoning over long sequence) + everybody is modeling wrong thing for this to work
@giffmana People who pick "MoEs are elegant" need to explain themselves lol.
MoEs are non-differentiable monsters.
@giffmana MoEs are like printing in parts and gluing them together. Works fine. Just not elegant.