The cynical thesis for automated research is that if you remove the human researcher, you can utilize the cluster better. Fable/GPT-5.6 can be trained to maximize information gain with grid search. Smaller experiments, more dynamic, without perf confounders. Better Than Taste.

*Claude devote 5% of the budget to optimizing kernels for every arch experiment before checking how it scales. make no mistakes* This is how you go from "oh noes DeepSeek is OOMs cheaper, necessity is the mother of invention" peasant morality fable to alien tech in months

@tszzl i want to believe

very hard to forecast (and credit-assign) these kinds of breakthroughs

and even if models can conjecture 100s of plausible breakthroughs, they may still require significant compute to derisk at scale

@tszzl ilya is intelligent and gets it, though he’s unlikely to find something. karpathy is a signal amplifier and doesn’t get it as indicated early by his app layer startup.

it will take *everyone* on the same page to get there. insane IPO targets and crash will accelerate this

@tszzl 100%

have they not figured out how to use ai to do ai research with enough of a loss factor to cut the hallucinations out (of the research mechanics)

i mean it would do this very fast so couldn't you do it with like 99% slop-loss and skim progress

nearly at the rate of diminishing returns we're now getting from manual grind

@teortaxesTex Unironically though

@tszzl And AI-driven advances in mathematical research may loop back around here. Humans just aren’t smart enough to keep this all in mind as that worry about Twitter clout and SF housing bidding wars.

markets like guaranteed scaling gains even if it's an OOM more capital for each breakthrough, but that also means the incentive for "cheap" breakthroughs increases relatively as well, so i think there's a good chance we'll get breakthroughs before RSI

(this also hinges on RSI not being here within a few months lol)

@tszzl Claude requiem 😂 captured the moment in one word

@tszzl Sounds like you’re bullish on fast takeoff

@tszzl

@tszzl I'm always skeptical of small model arguments though. It seems like they'll always need to have entirety of its knowledge of the world as well as it's reasoning skills compressed into the weights themselves. Eventually (maybe already) we are going to run into fundamental limits

@tszzl Given how energy efficient per intelligence the human brain is compared to the tech it has created, I think there are algorithmic gains on the table Idk how much there is in deep learning because of data scarcity: evolution took billions of years, but the internet is all eaten up

@tszzl Everyone's chasing the scaling laws these days, but not what actually makes them scale

@tszzl absolutely correct. i'm seeing great numbers as just some random dude with a laptop

@tszzl do you think a modal ai researcher type has also changed, i.e. the human type?

@tszzl Forward-deployed scientists will emerge: superstar polymathic researchers who triage disparate science in ways that create paradigm-shifting leaps equivalent to what the transformer was to AI, more often across every field. The transformer will be considered low-hanging fruit.

@teortaxesTex I think most models get stuck in an autoresearch local optimum if the loop is naive or there isn't a human. LLM's seem to be too sequential to intuit some sorts of common sense.

They often get stuck in the tactical and don't backtrack to the strategic.

@tszzl Not sure anything will top the wonder I felt first learning about the Universal Approximation Theorem and feeling like neural nets were capable of everything