A user says ideas from Nick Cammarata’s interpretability essay inspired the work, which is still in progress.
A user says they turned a couple of ideas from Nick Cammarata’s essay “Neural Networks Are Already Clean” into speedups for pretraining without backpropagation. They report a 2.3x speedup and say the work is still in progress.
—
2 Sources, first seen ago
Inspired by Nick Cammarata’s interpretability essay “Neural Networks Are Already Clean”, I turned a couple of its ideas into speedups for pretraining without backpropagation. Still in the works. 2.3x speedup, cool
Replacing backprop with zeroth-order optimization is inevitable. Introducing Noise: a speedrun for pretraining without backprop. Last week we presented Dust, the first method to approach backprop with SGD for pretraining transformers. We think another 1,000x improvement on top…
Not enough discussion yet.
No sentiment analysis available yet.
Not enough discussion yet.
No sentiment analysis available yet.
—
Not ranked yet
—
Not ranked yet
