Researcher Shares Paper on Token Order Prediction
Thread by ML researcher @prompt_Tunes discusses paper on transformer training objectives.
TLDR
ML researcher Andy posted a paper reading thread from his account @prompt_Tunes while working at @rumik_ai. The post states the paper suggests exact MTP may be too hard as an auxiliary loss. Instead it proposes predicting the order of upcoming tokens works better. The thread includes a photo link and covers training objectives for transformer models. Andy noted the paper had been on his reading list for some time before he reviewed it.
Combined views
6.5K
1 Source, first seen ago
Sources
AN 
Andy@prompt_Tunespaper reading thread. this one has been on my list for quite a while, finally got the time to give it a read! this paper suggests that maybe exact MTP is too hard as an auxiliary loss. instead predicting the order of the upcoming tokens works better.
- likes: 111
- replies: 3
- bookmarks: 80
- reposts: 19