2h ago

ML Engineer Suggests Training Networks On GPUs And TPUs Simultaneously

——0——
Original post
OPCharles 🎉 Frye @ MLSys#863Charles 🎉 Frye @ MLSys@CHARLES_IRL

one simple, but insane, version of this: train on both GPUs and TPUs, sum the gradients

12:50 PM · May 22, 2026 View on X
REPLYCharles 🎉 Frye @ MLSys#863Charles 🎉 Frye @ MLSys@CHARLES_IRL

it's plausible that this can be solved after pretrain, but exploring the sub-level sets can be pretty hard (they are disconnected, and id expect numerics to be one of the biggest diffs across basins)

Charles 🎉 Frye @ MLSysCharles 🎉 Frye @ MLSys@charles_irl

one simple, but insane, version of this: train on both GPUs and TPUs, sum the gradients

7:50 PM · May 22, 2026 · 238 Views
7:50 PM · May 22, 2026 · 240 Views