ML Progress Stems From Composing Existing Math Objects
Research engineer kalomaze argues across replies that ML advances come from composing existing math objects.
kalomaze, a research engineer at Prime Intellect, stated that machine learning progress arises from composing existing math objects rather than out-of-distribution truths like general relativity in physics. He added that what counts as progress shifts with GPU capabilities and that breakthroughs often break unstated assumptions. Omar Khattab, an MIT professor, replied that nearly all human inventions share this property. kalomaze further suggested that language models reveal latent structure in speech as sufficient raw material for continued progress.
sometimes extraordinary things exist out of distribution in a way where no existing composition can express it. this was true of physics with GR. i sincerely don't think this is true of machine learning where the underlying value in the math objects falls out of their composition
ML Progress Stems From Composing Existing Math Objects
Research engineer kalomaze argues across replies that ML advances come from composing existing math objects.
kalomaze, a research engineer at Prime Intellect, stated that machine learning progress arises from composing existing math objects rather than out-of-distribution truths like general relativity in physics. He added that what counts as progress shifts with GPU capabilities and that breakthroughs often break unstated assumptions. Omar Khattab, an MIT professor, replied that nearly all human inventions share this property. kalomaze further suggested that language models reveal latent structure in speech as sufficient raw material for continued progress.
Reactions
6 postsKA 
kalomaze@kalomazephysics has underlying truths. "what you can approximately and tractably do with [current GPUs] that results in a new macro category of thing being possible altogether" is uhhhh, a moving object. what people want out of the moving object shifts and varies along contingent axes...
- likes: 14
- replies: 1
- bookmarks: 2
- reposts: 0
KA kalomaze@kalomazei would like to exaggerate the "thing that looks like a composition of existing ideas, but lets you do a whole new category of thing" part specifically. it requires you to try to evaluate the space of what compositions are novel in terms of their macro outcomes. which is hard!
- likes: 9
- replies: 2
- bookmarks: 1
- reposts: 0
KA kalomaze@kalomazei think what has looked like "breakthroughs" so far are things that have broke ambient, unopinionated, & unstated assumptions, not necessarily those people argue about this is rare bc its HARD to think about the shadow of assumed constraints you're implicitly routing around
- likes: 9
- replies: 2
- bookmarks: 1
- reposts: 0
KA kalomaze@kalomazei think if language models have taught the world anything, it's that the latent shadow of structure that exists in what we say is, as raw material, some kind of sufficient for continued progress as we know it. what happens when you take that more seriously as a direct claim...?
- likes: 12
- replies: 2
- bookmarks: 3
- reposts: 0
OK Omar Khattab@lateinteraction@kalomaze I like this take but I feel resistant to the contrast against physics and GR. That is imo mostly how things look in very late reconstruction later. Imo there are probably close to zero human inventions that don’t have the property you’re ascribing to ML, and that’s…
- likes: 7
- replies: 1
- bookmarks: 4
- reposts: 0
KA kalomaze@kalomaze@lateinteraction i mean yes at some level everything can be described as search over compositions but, that's not a useful level of abstraction to describe my thoughts wrt "i don't think we're going to need truly exotic ideas at the 'new abstraction changes everything we…
- likes: 1
- replies: 0
- bookmarks: 0
- reposts: 0