Based on 18 visible X reactions from 33 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@willdepue "works across... optimizers" No, it doesn't, not really. With Muon, appropiate learning rates and long enough training, watermarking should get erased. In practical setups it might happen that Muon does notable damage but doesn't fully erase. https://docs.modula.systems/examples/weight-erasure/
@willdepue This stuff makes me think we are nowhere near optimal efficiency. We must be massively off course in some fundamental way, that these survive through training. Wonder if distilling from a teacher can preserve them in the student.
@willdepue That is a striking demonstration of how initialization can leave a persistent signature even after training. It also makes model provenance and hidden watermarking feel much more concrete.
@willdepue Oh...oh...this gives me all sorts of ideas lol
@willdepue "works across... optimizers" No, it doesn't, not really. With Muon, appropiate learning rates and long enough training, watermarking should get erased. In practical setups it might happen that Muon does notable damage but doesn't fully erase. https://docs.modula.systems/examples/weight-erasure/
@willdepue This stuff makes me think we are nowhere near optimal efficiency. We must be massively off course in some fundamental way, that these survive through training. Wonder if distilling from a teacher can preserve them in the student.
@willdepue That is a striking demonstration of how initialization can leave a persistent signature even after training. It also makes model provenance and hidden watermarking feel much more concrete.
@willdepue Oh...oh...this gives me all sorts of ideas lol
@willdepue Wait that's actually sick
@mayukh091 banger
kinda funny you can draw a smiley face in your neural net before training and it’ll be there afterwards you can also use photos. i trained a MNIST classifier initialized to my face and you can still see me at the end works across inits, weight decay, LR, optimizers, see below https://x.com/willdepue/status/2076581570782056523/photo/1
i survived tinystories training! (i need to boost init its too hard to see me from the start) https://x.com/willdepue/status/2076617090317078810/photo/1 https://twitter.com/willdepue/status/2076581570782056523
these are all super tiny runs, i'll run a real modded-nanogpt Muon with my face when my h200s free up (thanks to andromeda cluster for the free gpus btw!!)
@willdepue we should melt a ton of robots and humans hugging pictures inside to save us all.
my face https://x.com/willdepue/status/2076581611512971364/photo/1
gonna go train a gpt-2 model in a sec and see if my face can last in an MLP
Based on 18 visible X reactions from 33 accounts; directional sample.
Ask a question below.
Published answers will appear here.
@mayukh091 banger
kinda funny you can draw a smiley face in your neural net before training and it’ll be there afterwards you can also use photos. i trained a MNIST classifier initialized to my face and you can still see me at the end works across inits, weight decay, LR, optimizers, see below https://x.com/willdepue/status/2076581570782056523/photo/1
i survived tinystories training! (i need to boost init its too hard to see me from the start) https://x.com/willdepue/status/2076617090317078810/photo/1 https://twitter.com/willdepue/status/2076581570782056523
these are all super tiny runs, i'll run a real modded-nanogpt Muon with my face when my h200s free up (thanks to andromeda cluster for the free gpus btw!!)