Engram’s learned text patterns, from names to web boilerplate
SemiAnalysis says Engram helps a model reuse familiar patterns rather than reconstruct them. Its scan found names, code fragments, task instructions and website boilerplate.
TLDR
SemiAnalysis describes Engram as retrieving learned representations of short token sequences. Its scan found examples ranging from “Ian Goodfellow” to code fragments, copyright notices and licensing text.
SemiAnalysis highlights “pints of frozen yogurt” at layer 1 and “three times as many” at layer 14—objects versus a reusable relationship. It says different layers can use memory differently, and that Engram learns what helps predict text rather than what humans consider worth remembering.
Combined views
2.7K
2 Sources, first seen 5h ago
Engram’s learned text patterns, from names to web boilerplate
SemiAnalysis says Engram helps a model reuse familiar patterns rather than reconstruct them. Its scan found names, code fragments, task instructions and website boilerplate.
TLDR
SemiAnalysis describes Engram as retrieving learned representations of short token sequences. Its scan found examples ranging from “Ian Goodfellow” to code fragments, copyright notices and licensing text.
SemiAnalysis highlights “pints of frozen yogurt” at layer 1 and “three times as many” at layer 14—objects versus a reusable relationship. It says different layers can use memory differently, and that Engram learns what helps predict text rather than what humans consider worth remembering.