Pretraining On Massive Datasets Could Outperform RL Models Says Builder

(this post assumes Fable was mostly the result of a massive pretrain)

@VictorTaelin

Isn’t RL basically just trying to find solutions that aren’t present in your dataset, but in a very inefficient way? Or isn’t the purpose of neural nets in general to interpolate, which you only have to do with a lack of data?

So if you had a dataset covering all economically valuable tasks in every variation that should be preferable?

@VictorTaelin I totally disagree. RL is the ultimate way to reach super intelligence

One reason: Robots shouldn't look Human, it should look absolutely out of this world. It shouldn't resemble human. We write and express stuff in words because it's optimal for humans

AI should be alien to us

not gonna disagree with that but thinking is still important. maybe it should be more latent. we give it so many experts and so many of them to active per token, but some tokens are much harder than others. so maybe some dynamic routing size... idk how to set this up without getting shape errors. i guess looped transformer makes sense here.

damn i just answered my own question (facepalm)

@nicolasmelo I don't disagree with you!

RL itself is fine, but transformers can't do RL justice

@flowersslop that's exactly how I see RL (as layman)

@notselfmodel no hope of agi with transformers regardless

@VictorTaelin no hope of agi that way

@VictorTaelin This isnt as far off as you think. More diverse data in pre training makes RL so much more easy. Makes the difference between the model having to reason with fuzzy concepts it doesnt fully know about vs things its versed in.

@elliotarledge @VictorTaelin this right here was thinking

@VictorTaelin working with opus 4.8 feels like reward hacking

@VictorTaelin Actually a great movie that if you haven't watched you should go watch:

On this movie aliens do not resemble humans AT ALL But they are able to control and bend time through language

It's something we can't even comprehend with our current knowledge

AI should be more like that

By the way that fable thing was probably just 2 opus model one of them does other tries to find better way or looking for missing things kinda thing then merge 2 of them into a monolith chain of thought if you can merge these 2 persona well now you have almost same thing source: trumps hair

@VictorTaelin Why bother training a neural net at all. why dont we build a perfectly deterministic LLM exclusively using python elif's

@VictorTaelin You sound like @karpathy in 2022

@VictorTaelin with enough rl scale maybe you can just distill out an inefficient but general reasoning algorithm

@VictorTaelin we must have similar gut microbiomes cuz my ass is saying the same

@VictorTaelin Nah, because then we are constrained to form, you lose the possibility of new behavior emerging from training that were outside what we expected or produced.

Emergent behavior outside anthropomorphic structure is probably what will get us AGI.

@flowersslop @VictorTaelin Not going to be near as robust to perturbations (surprise!) from trying to interpolate over data alone. This matters when you're trying to be novel too. RL helps reasoning because the model learns to self-correct and this repairs it during/prepares it for offish le beaten path