Roland Memisevic reflects on kernel trick in AI

Roland Memisevic posted a reflection on the kernel trick as a former central breakthrough in AI that enabled convex optimization for model training instead of backpropagation and stochastic gradient descent. He noted conceptual overlaps with current techniques including self-attention and linear recurrent neural networks. The post was reposted by academic Kosta Derpanis and researcher Scott Reed.

Original post

SR#457@SCOTT_E_REED@ROLANDMEMISEVIC

RO

Roland Memisevic@ROLANDMEMISEVIC

It's hard to believe, but there was a time at which the kernel trick was considered the absolute holy grail of AI, as it made it possible to use convex optimization instead of back-prop and SGD. Incidentally, self-attention and linear RNNs feel weirdly reminiscent of those vibes.

9:18 AM · May 15, 2026 View on X