Users like the MTP training technique because it improves MoE model benchmarks and learning as shown in the DSV3 paper.
Based on 2 visible X reactions from 3 accounts; directional sample.
Ask a question below.
Published answers will appear here.
nice! i like the idea of MTP for better learning but i think the only study about the impact on model intelligence is in the dsv3 paper (and what i said previously about other models doing it only in mid-training kinda says that other didn't find huge gains in doing it from scratch)
@_xjdr @eliebakouch @zmkzmkz Agree
@teortaxesTex @_xjdr @zmkzmkz it's quite different MTP tho, i think they do it in parallel instead of sequentially iirc
@eliebakouch @_xjdr @zmkzmkz DSV3 paper itself cited Meta's original work
@eliebakouch @zmkzmkz mtp is really powerful, just not for inference
nice! i like the idea of MTP for better learning but i think the only study about the impact on model intelligence is in the dsv3 paper (and what i said previously about other models doing it only in mid-training kinda says that other didn't find huge gains in doing it from scratch)
@teortaxesTex @_xjdr @zmkzmkz it's quite different MTP tho, i think they do it in parallel instead of sequentially iirc
@eliebakouch @zmkzmkz mtp is really powerful, just not for inference
Users like the MTP training technique because it improves MoE model benchmarks and learning as shown in the DSV3 paper.
Based on 2 visible X reactions from 3 accounts; directional sample.
Ask a question below.
Published answers will appear here.