The methodology will anchor Choshen's new research laboratory.
@yoavartzi I am soon moving to the new lab, so there's a lot of thinking (and a big branch should be pretraining), but it is also already active. I think the most concrete pretraining challenge is With context
"It is time to separate language from language models" The revelation keeps bugging me, and while making the talk "multilingual?" I just gave. Thought I'd briefly share the contents of the talk:
@yoavartzi There's also the thought of what tricks are people doing post hoc, what if we pretrain? e.g. speculative decoding is ~ deepseek's next token prediction trick. Thinking and instructions there are paper that show it. What else do we do only at the end?
@yoavartzi I am soon moving to the new lab, so there's a lot of thinking (and a big branch should be pretraining), but it is also already active. I think the most concrete pretraining challenge is With context
@LChoshen This is cool. I wasn't aware of this line of work. Results look strong. Thanks!
@yoavartzi I am soon moving to the new lab, so there's a lot of thinking (and a big branch should be pretraining), but it is also already active. I think the most concrete pretraining challenge is With context
The methodology will anchor Choshen's new research laboratory.
@yoavartzi I am soon moving to the new lab, so there's a lot of thinking (and a big branch should be pretraining), but it is also already active. I think the most concrete pretraining challenge is With context
"It is time to separate language from language models" The revelation keeps bugging me, and while making the talk "multilingual?" I just gave. Thought I'd briefly share the contents of the talk: