Reaction
Synthetic gradients for regimes where naive backpropagation isn't tractable
A commenter finds evolution strategies and genetic algorithms uninteresting as alternatives to backpropagation. They point instead to synthetic gradients when naive backpropagation isn't tractable without backpropagation through time.
TLDR
The commenter says synthetic gradients, or DNI, can behave close to full backpropagation but lose to it where full backpropagation is tractable. They suggest a modern approach, especially a distributional one, may offer a fairly faithful approximation of an “unobservable regime” that cannot be trained directly.
Combined views
1.3K
6 Sources, first seen 7h ago