Fuzhao Xue Shares Looped MoE Transformer Work
Google DeepMind researcher reflects on school projects involving looped transformers and data efficiency.
Fuzhao Xue posted on X about research he conducted while in school. He described work on a looped MoE transformer and stated that looped models show better data efficiency. Xue, identified as a Senior Research Scientist at Google DeepMind working on Gemini pretraining, added that the ideas seem obvious and cheap though he remains proud of the projects. The post links two arXiv papers, one titled Go Wider Instead of Deeper and another titled To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis. The public posts contain no additional claims or updates from other sources.
Combined views
21.7K
2 posts, first seen 10h ago
Reactions
2 postsFX 
Fuzhao Xue (Frio)@XueFzWorked quite a bit on loop transformer when I was in school: Looped MoE transformer: Looped model has better data efficiency: https://arxiv.org/abs/2305.13230 But these ideas matter very little to be honest. They are obvious and cheap (still proud of worked on these projects…
- likes: 197
- replies: 3
- bookmarks: 151
- reposts: 15
WH wh@nrehiew_The main motivation is that it allows us to scale compute. Typically, we think of scaling as data, params and CoT. Looping allows us to increase compute/token while keeping param sizes small
- likes: 28
- replies: 1
- bookmarks: 5
- reposts: 3
