Elie Bakouch says Mistral AI co-founder Guillaume Lample's FP8 training run matches the learning rate schedule of DeepSeek-V2

@GuillaumeLample

this is the exact same lr schedule as deepseek v2 in terms of when the lr changes

the difference is that the second phase does seem to be a slight decay instead of constant, and of course cooldown is also a decay instead of constant

@GuillaumeLample Do not expose Le Chaton Fat to ozempic please (no quantization)

@GuillaumeLample @yacineMTB Now that's some proper cooldown

guillaume tweeting the loss curve also makes me hope that they release a tech report for the next big chaton model 🙏

@GuillaumeLample @ESYudkowsky prepare the ICBMs

@GuillaumeLample

@GuillaumeLample

@GuillaumeLample How far we’ve come

@GuillaumeLample let’s see paul allens loss curve

@GuillaumeLample qlq a une comparaison je me rends pas compte je connais rien en ml

@GuillaumeLample Cook cook cook

@GuillaumeLample Sudden drop!

@GuillaumeLample @grok ça veux dire quoi ?

@GuillaumeLample Is it finally done training?

@GuillaumeLample 😸

@GuillaumeLample It's happening !!!

@GuillaumeLample `run000` & `percent_done` 🤣

@GuillaumeLample Keep on training for a few 1000 iterations more. We can wait for a quality product.

@GuillaumeLample We will meme this into reality