Announcement
No-backprop transformer run reportedly sets an open-architecture FineWeb-30M record
A contributor reports 3.2115 validation loss, down from 3.4095, after one epoch on a single eight-H100 node.
TLDR
A Noise contributor reports a new open-architecture record on the FineWeb-30M zero-order optimization benchmark. The transformer reached 3.2115 validation loss without backpropagation, down from Noise’s previous 3.4095, in 2 hours 57 minutes. The track permits changes to both architecture and training; the new run used a larger model with hashed bigram, trigram and skip-bigram tables.
Combined views
—
3 Sources, first seen ago
