Announcement
SOAP and SPlus reportedly scale strongly to 96-plus DiLoCo workers
A researcher says whitening the update’s covariance enables that scaling, with 96 providers given as an example.
TLDR
A researcher says new work shows SOAP and SPlus achieve strong worker scaling to 96-plus DiLoCo workers by whitening the update’s covariance. They frame it as an option for frontier-model pre-training when no provider will sell enough co-located chips, giving 96 providers as an example.
Combined views
—
2 Sources, first seen ago
