Fleuret compared LLM understanding to abstract mathematical concepts.
So far, any "understanding of the world" a LLM has is an implicit by-product of its modeling of language.
The closest in a human mind is probably our model of abstract objects that have no reality but their textual description (mathematical objects, abstract physics)
1/3
No Digg Deeper questions have been answered for this story yet.
But contrary to the LLM, the human mind bootstraps its learning with a multimodal stream of a coherent world, language comes later as a symbolic and efficient compression of views of the resulting model.
2/3
So far, any "understanding of the world" a LLM has is an implicit by-product of its modeling of language.
The closest in a human mind is probably our model of abstract objects that have no reality but their textual description (mathematical objects, abstract physics)
1/3
Even the most abstract object the human brain produces is anchored into a world model that is not contingent on language.
An LLM's world model is the shadow of its language, a human language is the shadow of their world model.
3/3
But contrary to the LLM, the human mind bootstraps its learning with a multimodal stream of a coherent world, language comes later as a symbolic and efficient compression of views of the resulting model.
2/3
@francoisfleuret That's technically correct but also misleading. Our world model is a shadow of our perception, and the sheer amount of text going into LLMs is constraining this shadow more than the perceptual patterns a human brain can encounter in a lifetime of experience.
Even the most abstract object the human brain produces is anchored into a world model that is not contingent on language.
An LLM's world model is the shadow of its language, a human language is the shadow of their world model.
3/3
@Plinz More?
@francoisfleuret That's technically correct but also misleading. Our world model is a shadow of our perception, and the sheer amount of text going into LLMs is constraining this shadow more than the perceptual patterns a human brain can encounter in a lifetime of experience.
@francoisfleuret human intelligence is best with things "close to us". what is weird is that what is "close" to LLMs is abstract to us. Learning about the world backwards. I dont know that human language is only a compression of views though
An LLM's world model is the shadow of its language, a human language is the shadow of their world model.
Even the most abstract object the human brain produces is anchored into a world model that is not contingent on language.
@francoisfleuret it becomes interwoven with the other streams.
@francoisfleuret Biological minds needs constant sensory input to prevent hallucinations.
@francoisfleuret I'm wondering if multimodal training even fixes this. We still prioritize language first and bolt on vision later, which is backwards from how humans actually bootstrap learning.