Gwern anticipated AI scaling trends before nearly all observers outside OpenAI by concluding intelligence stems from compute, data, and parameters rather than novel algorithms.
Pedro Domingos cites historical cycles where scaling alone proved insufficient.
#70Dwarkesh Patel@DWARKESH_SPWatch the full interview for more:
Danny Hillis was scaling up AI with a massively parallel supercomputer in the 80s. In the 90s we had the data mining explosion, a.k.a. scaling up ML. In the 2000s we had the "big data" boom. And each time we noticed that no, compute etc. is not enough - you really need better algorithms. Too bad we're doomed to repeat history.
Dwarkesh Patel@dwarkesh_sp.@gwern saw AI scaling coming before almost anyone outside OpenAI. How was he able to predict this trend? He told me that his core idea was that intelligence is just compute, data, and parameters - no clever algorithm needed. He didn't come to this view in a eureka moment. He just read paper after paper and nudged his priors a little each time, until the trend seemed obvious.