Andrew Trask argues LLMs cannot learn information that was not first encoded as a gradient during backpropagation

@iamtrask Exactly....Gradients aren’t some mysterious extra thingthey’re literally the only way the model ever learns anything meaningful. If they didn’t end up with semantic structure, backprop would just be moving noise around for no reason. 👾👀 who agrees👇🏻