The project revives Evolution Strategies to bypass gradient-based training.
Over the past 90 days I dropped everything to study Zeroth-Order Optimization. Forgotten techniques like Evolution Strategies, which can train models without backprop.
I also wrote multiple codebases. ZOTitan for training, a kernel autoresearch harness, and kernels.
Users praise the writeup on Zeroth-Order Optimization and ZOTitan for its depth and value that rewards multiple readings.
No Digg Deeper questions have been answered for this story yet.
Transformers train very well with backprop. But at long contexts things start to fall apart.
RNNs scale better to long contexts in theory, but it is numerically impossible to backprop through infinite context lengths.
But what if we didn't need backprop? What if It just worked?
And lastly, you can find the article here. https://apaz.dev/blog/Scaling_To_Unfathomable_Depth.html
banger article alert
Over the past 90 days I dropped everything to study Zeroth-Order Optimization. Forgotten techniques like Evolution Strategies, which can train models without backprop.
I also wrote multiple codebases. ZOTitan for training, a kernel autoresearch harness, and kernels.
You can find my training code here https://github.com/apaz-cli/ZOTitan
Evolution Strategies was explored as an alternative to RL by OpenAI in 2017, but nobody thought it would scale to training LLMs because the signal-to-noise ratio gets worse with the number of parameters. That's really bad. So for a while no one scaled it.
Counterpoint. It works.
I wrote a really long article. Part blog post, part math dump deriving why zeroth-order methods supposedly suck, part research notes and idea vomit. Many of these ideas should be papers and probably will be at some point.
You can find the kernel autoresearch harness I wrote here https://github.com/apaz-cli/kernelthing
You can find MeZO example kernels here https://github.com/apaz-cli/MeZOKernelExample
Most of what we know about deep learning has to be re-evaluated or at least doublechecked under a new paradigm like this. The architecture search space just got blown open, the optimal optimizer probably looks really different, and there are tons of new scaling laws to derive.
I think the path forward is to automate more and more of the research. There is just too much to do, and it is too important not to do. Incidentally, Zeroth-Order approaches lend themselves very well to continual learning, under ZO training and inference are one and the same.
Thanks to @ariaurelium and @snowclipsed for proofreading, and to @antferdom/@verdacloud for the compute for running experiments.
@apaz_cli This was a really cool write up! Explored a lot of things we’ve been looking at. (Team behind es at scale)
@apaz_cli really nice writeup, feels like the type of thing i can read a few times and get something new out of it each time
