MIT Professor Challenges Pure DNNs on Scaling Behavior
MIT professor Omar Khattab states frontier models add non-differentiable parts to standard networks.
Omar Khattab posted that current frontier systems are not pure differentiable DNNs. He noted they include scratchpads, offloaded context, PTC, and recursion, which let them exceed plain GPT-4 with RLHF on functional tasks. Susan Zhang shared a screenshot of Yann LeCun quoting an earlier critique of LLMs as an AGI offramp. Other replies discussed whether past critics have shifted positions without acknowledgment. Khattab urged keeping hypotheses clear between conventional DNNs and augmented setups. The thread stays on technical definitions rather than broader claims.
Nicely, this is entirely falsifable. You think DNNs with differentiable operations are all you need? Fantastic - just show the same scaling and generalization behavior with one. Even on purely functional tasks that don’t inherently involve a tool env. Only text in, text out.
Reactions
11 postsOK 
Omar Khattab@lateinteractionTPOT loves to make fun of LeCun and Marcus lately for the sin of still criticizing "pure LLMs" in 2026. But a scratchpad-equipped Transformer with offloaded context, PTC, and recursion is unequivocally no longer a pure DNN, and that's why it exceeds plain old GPT-4 with RL[HF]!
- likes: 43
- replies: 2
- bookmarks: 11
- reposts: 3
OK Omar Khattab@lateinteractionwhether that means any specific position has been wrong/right "all along" is a separate topic, especially if such a position was not constructive (as in "here's how to do this") but let's not muddle up our hypotheses classes please; a conventional DNN is a very clear object
- likes: 23
- replies: 0
- bookmarks: 3
- reposts: 0
XL xlr8harder@xlr8harder@lateinteraction The criticism is that they have retreated to "pure LLMs" without acknowledging the retreat. People who said LLMs were a mistake are now claiming that what they meant all along was that LLMs were only part of the solution.
- likes: 3
- replies: 1
- bookmarks: 0
- reposts: 0
OK Omar Khattab@lateinteractionhaha i'm only OCD'ing on a technical topic: are RLMs purely neural or not, and is the evidence currently in favor of pure DNNs or not. i really think two things are true: (1) the current frontier are not pure differentiable DNNs, obviously, and (2) LLMs are absolutely the…
- likes: 4
- replies: 1
- bookmarks: 1
- reposts: 0
XL xlr8harder@xlr8harder@lateinteraction fair, and i appreciated the thread!
- likes: 2
- replies: 0
- bookmarks: 0
- reposts: 0
SZ Susan Zhang@suchenzangthis is how they all come back and rewrite history "i was right all along" sure you were, gramps, sure you were but grannie remembers
- likes: 801
- replies: 25
- bookmarks: 96
- reposts: 27