Jie Tang on Scaling Laws Beyond Parameters
Professor highlights role of data, compute allocation, and inference in model performance.
Jie Tang, Tsinghua professor and Zhipu AI co-founder, posted that parameter count alone does not answer how capable a model will be. He stated it must be weighed with data volume, compute allocation choices, and inference demands. The post drew replies from Nando de Freitas, Thomas Wolf, Julian Togelius, Delip Rao, and others who described it as a clear account of current scaling trade-offs and a masterclass on the subject.
Combined views
1.3M
13 posts, first seen 5d ago
Reactions from ranked influencers
13 posts
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have,…
«Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim»
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have,…
I recommend meditating on this. Prof. Tang is obviously correct and I've been saying the same thing for a while. The question is, what do different scaling axes buy you? Why can V4-Flash get 61% on ARC-AGI 2? Why does GLM 5.3 perform so well? Lisan-tier scaling 101 is shallow.
Interesting take by the zAI (GLM-Models) founder: AI scaling is not over, but we just focused too much on model size. For years, the industry mostly asked: How many parameters does a model have? But performance also depends on training data, inference compute and post-training.…
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have,…
This is a masterclass in scaling laws, and also on how to share research outcomes.
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have,…
I wish every neolab had a professor as cofounder of the level of @jietang and so able to put in perspective their new model release. A great snapshot on the history of scaling laws
Thoughts About Scaling Law Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have,…