Lukasz Kaiser, OpenAI senior research scientist and 2017 Transformer paper co-author, defended current architectures in a live 80-minute debate on Transformer versus post-Transformer designs

One of my bigger gripes with LeCun-style reasoning is that they will endlessly hype up cat-intelligence and inherent superiority of biological brains and limits of Transformers/other mainstream DL, but it's all atomic proofs at best. Clearly, *stacked* chairs are Turing-complete.

This was so much fun!!

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion @zuzanna_pathway I can't wait for you to release the whole thing. What a great event! Congratulations! 💪👍

@ShashankDe5535 > world modelling requires predicting higher level features than token/pixel supervision allows well I don't see evidence that "token/pixel supervision", depending on the objective, precludes learning of arbitrarily complex features

@teortaxesTex I don't LeCun's arguments. But the realistic scenario to me is that RL doesn't generalize. And anything we can't sim or have otherwise an objective and cheap eval function for will just never happen short of evo search brute force (writting for example)

@teortaxesTex This is a bit misleading. Lecun does not have anything against transformers or mainstream DL. It is the canonical architecture also in the JEPA pipelines.

Lecun's central claim is that world modelling requires predicting higher level features than token/pixel supervision allows

@KinasRemek @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion Coming soon :) there will be many more boxing moves :D

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion Can't wait for the full video!

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion SF needs more of such cool AI events!

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion Quality execution! Loved to watch this one

@teortaxesTex Dog-like intelligence explains it better to me. My dog continuously digs and circles when lying down on the sofa or bed, as if bedding down in a wild meadow. What should we expect from JEPA’s priors on situational awareness then?

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion 🤩

@teortaxesTex The evidence today is stacked quite clearly against Lecun, there is no denying that

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion the fact you got the actual inventors debating says more than any benchmark. most vs content is marketing noise, this forces real technical accountability.

@teortaxesTex You can grow brain organoids and probably just grow big brains like artifical meat in big factories if thats necessary lol

@zuzanna_pathway @lukaszkaiser @adrian_pathway @mlech26l @YesThisIsLion Five minutes. Pick your side. 🥊

@teortaxesTex stacked chairs being turing complete is the kind of analogy that proves his point more than he realizes

the real ceiling isnt transformers, its our will to stack them right