A user asks what remains mysterious about generalization; a reply points to defining “domain” and distinguishing generalization from memorization.
A user asks which questions about generalization remain open or mysterious. In reply, another user says clearer definitions are a starting point: What counts as a “domain,” and what does it mean to generalize rather than memorize?
23.1K
9 Sources, first seen ago
—
Not ranked yet
—
Not ranked yet
@yoavgo@andrewgwils proper definitions for a start. what is "a domain"? what does it mean to generalize vs memorize?
@tdietterich@andrewgwils The kind of generalization I would like to have: Given (x,t) pairs for an object under constant acceleration a, figure out that x = 1/2 a t^2, a rule that generalizes universally. Q2: Do LLMs do this? Or are they just doing kernel interpolation on the pairs?
@ylecun@tdietterich @andrewgwils We would have to insist that we prefer models that can be expressed with a small number of short formula, using common mathematical primitives. That's an inductive bias and/or regularizers that symbolic regression employ. But then, why do we think this is good?@yoavgo@andrewgwils proper definitions for a start. what is "a domain"? what does it mean to generalize vs memorize?@tdietterich@andrewgwils The kind of generalization I would like to have: Given (x,t) pairs for an object under constant acceleration a, figure out that x = 1/2 a t^2, a rule that generalizes universally. Q2: Do LLMs do this? Or are they just doing kernel interpolation on the pairs?@ylecun@tdietterich @andrewgwils We would have to insist that we prefer models that can be expressed with a small number of short formula, using common mathematical primitives. That's an inductive bias and/or regularizers that symbolic regression employ. But then, why do we think this is good?@yoavgo@andrewgwils proper definitions for a start. what is "a domain"? what does it mean to generalize vs memorize?@tdietterich@andrewgwils The kind of generalization I would like to have: Given (x,t) pairs for an object under constant acceleration a, figure out that x = 1/2 a t^2, a rule that generalizes universally. Q2: Do LLMs do this? Or are they just doing kernel interpolation on the pairs?@ylecun@tdietterich @andrewgwils We would have to insist that we prefer models that can be expressed with a small number of short formula, using common mathematical primitives. That's an inductive bias and/or regularizers that symbolic regression employ. But then, why do we think this is good?@yoavgo@andrewgwils proper definitions for a start. what is "a domain"? what does it mean to generalize vs memorize?@tdietterich@andrewgwils The kind of generalization I would like to have: Given (x,t) pairs for an object under constant acceleration a, figure out that x = 1/2 a t^2, a rule that generalizes universally. Q2: Do LLMs do this? Or are they just doing kernel interpolation on the pairs?@ylecun@tdietterich @andrewgwils We would have to insist that we prefer models that can be expressed with a small number of short formula, using common mathematical primitives. That's an inductive bias and/or regularizers that symbolic regression employ. But then, why do we think this is good?
