Report
Dust reportedly reached a loss below 4 within days
A contributor says Dust perturbs activations instead of weights, enabling associative-memory estimates and orthogonal noise draws.
TLDR
A contributor says their team got Dust below 4 loss within a few days. They credit two techniques for the zeroth-order optimization work: using past inputs and gradients to make rough estimates that reduce update variance, and orthogonal noise draws to avoid probing the same direction twice. The contributor says Dust’s smaller activation space makes those draws practical.
Combined views
37
1 Source, first seen ago
