Reaction
Synthetic pretraining may affect how AI model features overlap
A user says Baguettotron's feature connections differ sharply from Gemma-270m's, perhaps more because of synthetic pretraining than deep architecture.
TLDR
A user argues that training-data shape affects how AI model features overlap, citing what they describe as extensive “clean” synthetic training data for Qwen-3 and Kimi. They say their analysis found Baguettotron's “commit zone” topology radically different from Gemma-270m's, with many connections between features. They suspect synthetic pretraining may explain the difference more than deep architecture.
Combined views
—
1 Source, first seen ago
