Many accounts welcomed Dust as an exciting zeroth-order alternative to backprop for pretraining transformers, while negative replies dismissed the early experiments and benchmarks as unreasonable or unconvincing.
The team claims Dust can approach and sometimes exceed backpropagation when given large amounts of computation. It also says the method is roughly 1,000 to 10,000 times more compute-efficient than EGGROLL for training transformers. The comparison applies specifically to the team's reported transformer-training results.
A virtual population of perturbations
The central idea is what the researchers call a "virtual population," intended to scale to large populations by perturbing activations in parallel.
The team says backprop-like gradients emerge from a large population of those perturbations. It reports that the alignment held at every scale tested, up to 1 billion training tokens.
The researchers also report a counterintuitive scaling result: At a fixed population size, larger models, including models up to 120 times larger in their tests, reached lower loss than smaller ones. They present that pattern as evidence that zeroth-order search does not necessarily become less effective as model size grows.
The compute comparison is contested
The early reaction includes a significant caveat. Sadhika Malladi questioned whether the results amount to meaningful pretraining, pointing to a FineWeb validation loss of about 5.
Malladi also argued that plots against token count do not provide the most useful comparison. She suggested steps or GPU-hours instead, which would make the computational cost of the approach easier to judge.
Her criticism leaves the central comparison unsettled: how much computation Dust spends to reach a given loss.
Many accounts welcomed Dust as an exciting zeroth-order alternative to backprop for pretraining transformers, while negative replies dismissed the early experiments and benchmarks as unreasonable or unconvincing.
