Reactions from ranked influencers
2 posts
There are so many ideas to improve learning that are worth exploring
Real brains follow Dale's principle: a neuron can either excite its neighbors or suppress them, but never both. Standard deep learning ignores this and uses backpropagation. In our new paper, Diffusing Blame, we fix this disconnect. By introducing a routing method that broadcasts error signals directly to the hidden layers, we can train networks made of dedicated positive and negative neurons to strictly obey Dale's principle, all without relying on backprop! This method works surprisingly well on image recognition tasks despite the strict biological constraints. We also achieved competitive, backprop-free reinforcement learning on complex locomotion tasks and the open-ended Craftax environment. It is neat to see that representation learning remains possible even when we force deep learning to play by the rules of real neurons.
@chris_j_paxton Yes, though the statement that "brains follow Dale's law" isn't really true (if anything it's a massive simplification, not unlike ANN units! 🙂)