Some users like the idea of MTP for better learning in models.
Based on 1 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.
nice! i like the idea of MTP for better learning but i think the only study about the impact on model intelligence is in the dsv3 paper (and what i said previously about other models doing it only in mid-training kinda says that other didn't find huge gains in doing it from scratch)
@eliebakouch @zmkzmkz i should have qualified, i use MTP like architectures a lot (and for a lot of different things), and in my experience, they are very powerful. i have not found them particularly useful for speeding up inference however
@_xjdr @zmkzmkz hmm i'm not sure we have proof of this, lot of model like glm or stepfun or longcat only do it in midtraining to get the inference gain
nice! i like the idea of MTP for better learning but i think the only study about the impact on model intelligence is in the dsv3 paper (and what i said previously about other models doing it only in mid-training kinda says that other didn't find huge gains in doing it from scratch)
@eliebakouch @zmkzmkz i should have qualified, i use MTP like architectures a lot (and for a lot of different things), and in my experience, they are very powerful. i have not found them particularly useful for speeding up inference however
@_xjdr @zmkzmkz hmm i'm not sure we have proof of this, lot of model like glm or stepfun or longcat only do it in midtraining to get the inference gain
Some users like the idea of MTP for better learning in models.
Based on 1 visible X reactions from 2 accounts; directional sample.
Ask a question below.
Published answers will appear here.