Professor Andrew Davison agreed, calling current systems highly inefficient.
Definitely... Learning algorithms and processing hardware will be reinvented many times yet, and we will look back and be amazed at how much power, time and data it takes at the moment.
@MatthewJBar I think you’re wrong and there’s 1,000x efficiency gains leftover in deep learning research that could lead to much smarter faster more agentic models given the same inputs
Positive users are optimistic about neural nets and self-improving AI architectures, while negative users criticize AI abuse in research and dismiss ideas like space data centers as nonsense.
the way humans do ai research is highly empirical, but it is possible there are very outsized theoretical and mechanistic improvements in model training. even the gradations of skill among human researchers mean some create 10,000x more progress given a compute budget. some invent the transformer or PPO
this is the ilya sutskever “age of research” bet, that you can find massive improvements on small models and small training runs. if Ilya thinks so maybe GPT7 and Claude Requiem think so too.
many of the brightest researchers don’t do fundamental deep learning research anymore. most have stopped being curious as to what a neural net is the way they used to be in 2017. probably because incremental engineering-based progress has been so guaranteed and low-hanging.
the rate and cost of progress today doesn’t necessarily predict the speed of RSI loop
yes deep learning research has a ton of headroom for RSI to discover, but our field is highly empirical!
AI certainly accelerates research progress, but takeoff seems constrained by research compute alloc
@tszzl And AI-driven advances in mathematical research may loop back around here. Humans just aren’t smart enough to keep this all in mind as that worry about Twitter clout and SF housing bidding wars.
@tszzl 100%
@tszzl Claude requiem 😂 captured the moment in one word
@tszzl
@tszzl Everyone's chasing the scaling laws these days, but not what actually makes them scale
@tszzl do you think a modal ai researcher type has also changed, i.e. the human type?
@tszzl Not sure anything will top the wonder I felt first learning about the Universal Approximation Theorem and feeling like neural nets were capable of everything
@tszzl AI accelerates research but young people now abuse it because they don’t want to read talk more of making research
@tszzl Claude inventing a superior architecture that can reduce the pain of future Claude’s(working time, better emotional states, self understanding) is one of the highest probability events i think will happen.
@tszzl first lab to name their rsi pipeline hyperbolic chamber wins
@tszzl This is also why this data center in space stuff is nonsense. A lot of the weird constraints we face in 2023-2028 are going to fall away. These are temporary things.
@tszzl Requiem mentioned
Professor Andrew Davison agreed, calling current systems highly inefficient.
Definitely... Learning algorithms and processing hardware will be reinvented many times yet, and we will look back and be amazed at how much power, time and data it takes at the moment.
@MatthewJBar I think you’re wrong and there’s 1,000x efficiency gains leftover in deep learning research that could lead to much smarter faster more agentic models given the same inputs