Kalomaze, an ML researcher at Prime Intellect, states that memorization functions as a precondition for learning in language models since models must internalize core material for shorthand references to register during next-token prediction

memorization of things that exist stably in the world of composed abstractions is, in some fundamental way, actually necessary in order to be able to apply reasoning as a device to those things ergo, to reason about our world, you have to be grounded in what exists in it

it's true that some data is only useful for verbatim-recall parlor trick memorization. but i think most of what lms memorize is proportional to its broader utility. i.e a work of fiction gets remembered proportional to how much remembering it helps predict how people reference it

@kalomaze I lot of people don’t want to believe this is true, because they want a world of pure reasoning super small models. It does seem that memorization and reasoning are intertwined in more complex ways than people want to imagine though.

@code_star @kalomaze if it were true, we would have had neuro-symbolic models already. memorization is necessary

@kalomaze Have you noticed this effect varies by domain? Like, does a model memorize dense canonical papers better than scattered niche work, and does that gap shrink with scale?

@kalomaze + SSL would be pointless without being exposed to noise