the way humans do ai research is highly empirical, but it is possible there are very outsized theoretical and mechanistic improvements in model training. even the gradations of skill among human researchers mean some create 10,000x more progress given a compute budget. some invent the transformer or PPO

this is the ilya sutskever “age of research” bet, that you can find massive improvements on small models and small training runs. if Ilya thinks so maybe GPT7 and Claude Requiem think so too.

many of the brightest researchers don’t do fundamental deep learning research anymore. most have stopped being curious as to what a neural net is the way they used to be in 2017. probably because incremental engineering-based progress has been so guaranteed and low-hanging.

the rate and cost of progress today doesn’t necessarily predict the speed of RSI loop

the way humans do ai research is highly empirical, but it is possible there are very outsized theoretical and mechanistic improvements in model training. even the gradations of skill among human researchers mean some create 10,000x more progress given a compute budget. some invent the transformer or PPO

this is the ilya sutskever “age of research” bet, that you can find massive improvements on small models and small training runs. if Ilya thinks so maybe GPT7 and Claude Requiem think so too.

many of the brightest researchers don’t do fundamental deep learning research anymore. most have stopped being curious as to what a neural net is the way they used to be in 2017. probably because incremental engineering-based progress has been so guaranteed and low-hanging.

the rate and cost of progress today doesn’t necessarily predict the speed of RSI loop