Oriol Vinyals says co-author Wojciech Zaremba wanted to retract their seminal RNN Regularization paper a year after publication

Fun fact: @woj_zaremba wanted to completely remove that paper from existence a year or so after. I forgot why though -- Woj?

@OriolVinyalsML @woj_zaremba I'm really grateful it exited, the first 3-6 months of my career was just reading that code repeatedly and trying to improve it, I'd finish my day by running experiments on the 2-4 GPUs I had and then I'd wake up, look at results and start new experiments.

@OriolVinyalsML @woj_zaremba I'm also so grateful Oriol for your support of my early work on the ML subreddit, every time I'd post a paper in those first few years you left a supportive comment. There weren't a lot of people doing LM research in those days so it really meant a lot to hear from you!

@OfirPress @woj_zaremba @OriolVinyalsML @ilyasut ``` weight tying solved something real.

it's kinda wild how long it stuck around even after transformers changed everything else

@OfirPress @woj_zaremba @OriolVinyalsML @ilyasut 10 years of improving perplexity on a 1M token dataset is wild context for a technique that still gets used

@OfirPress @woj_zaremba @OriolVinyalsML @ilyasut Weight tying was cool for a week back in 2017. Whatever.