The bitter lesson says that the thing that matters is finding the right thing to scale up. It does not say that scaling up the wrong recipe will always beat a slightly less scaled up, but more correct recipe.
One reflection at the end. It is common (especially on Twitter) to use the bitter lesson to argue that whoever owns the FLOPs will win. But this is an incorrect interpretation of the bitter lesson.
Users enjoyed the researcher's clarification of the Bitter Lesson on AI scaling and compute because it highlighted the value of algorithmic advances found through personal projects.
No Digg Deeper questions have been answered for this story yet.
Overall, I quite enjoyed doing this project while funemployed! The overarching lesson for me was that algorithmic advances (if you can find them) are a very, very real thing. And as fast as progress in AI is now, there is a very real chance that things may go even faster soon.
To be more precise, the bitter lesson claims that FLOPs are necessary but not sufficient alone.
Full blog post, including links to the model weights for the SFT / RL GPT-2 checkpoints, here, along with a few additional experiments / comments.
https://hughbzhang.com/blog/rl-on-cot-gpt2-era.html
To be more precise, the bitter lesson claims that FLOPs are necessary but not sufficient alone.
There are many examples of companies that held large compute leads and did not succeed in training good models. Why? Because they did not scale the right way.
There are many examples of companies that held large compute leads and did not succeed in training good models. Why? Because they did not scale the right way.
The bitter lesson says that the thing that matters is finding the right thing to scale up. It does not say that scaling up the wrong recipe will always beat a slightly less scaled up, but more correct recipe.