Lilian Weng, co-founder of Thinking Machines Lab, analyzes scaling laws to reconcile Kaplan and Chinchilla models

I wonder how many more people will just ask model to summarize the post, in comparison to a year ago. 😂

@lilianweng @TheZachMueller would be very cool to see how kaplan/chinchilla compare to much more recent work from other labs, Farseer comes to mind (https://arxiv.org/abs/2506.10972)

Soon I should set up the model to update Lil’Log automatically but I’m not there yet.

@lilianweng In our work -- https://arxiv.org/pdf/2605.29448 We formally show the common theme connecting all the scaling laws -- Submodularity and Diminishing Returns.

@lilianweng what a timing. recently had to go through a few scaling laws papers, seemed interesting but then foundational works(especially Chinchilla) seemed too overwhelming to go through properly. Will give it a read. Thanks.

@lilianweng somehow you always make a blog post when im getting much deeper into a topic, still remember reading your diffusion one when we were making dreambooth (thank you!). and now this one

@lilianweng 😭

@lilianweng Thanks for sharing! Hope all’s well with you.

@lilianweng Nah, your blog posts are the stuff we read through (and dedicate half the afternoon)

@lilianweng love your writing! your posts have been a real joy to read over the years and they genuinely helped me get through my phd.

@lilianweng I'd be very careful asking a model to summarize this paper. I doubt most models really understand the math equations in it. 😂

@lilianweng I keep coming back to eval quality here. Bad evals make compute allocation look scientific when the target is fuzzy.

@lilianweng thanks for making my day!

@lilianweng I can do it for you Ms. Weng

@lilianweng This is one of the most concise yet comprehensive write-ups I have seen on the topic. Great job!

@lilianweng scaling laws feel boring until you need to burn millions on a run. then boring is exactly what you want.

@lilianweng Summarize not working 🤔