Lisan al Gaib agreed the stages cannot run simultaneously
@scaling01 @AndrewCurran_ this is literally what you need to do if you want to « scale », you can’t yolo a 10T model aha
@AndrewCurran_ i mean he said that, but is it true?
why would he train: 2 x 1T models 2 x 1.5T models a 6T model and a 10T model
all at the same time?
I don't think we are getting the 10T that soon
No Digg Deeper questions have been answered for this story yet.
@eliebakouch @AndrewCurran_ isn't each smaller run the proof for the next bigger one?
@scaling01 @AndrewCurran_ why not? if you have enough gpu to do in parallel this is more efficient
@eliebakouch @AndrewCurran_ Yeah but not all at the same time
@scaling01 @AndrewCurran_ this is literally what you need to do if you want to « scale », you can’t yolo a 10T model aha
@scaling01 @AndrewCurran_ why not? if you have enough gpu to do in parallel this is more efficient
it is, but i mean you can speed up the process by just doing multiple scales at once if you have idle nodes and are confident in your recipe
like for instance let's say you had established your loss scaling ladder and want to see if the prediction is accurate or not, you can launch both sizes at the same time instead of sequentially (of course if the smaller one fails, not a good sign for the bigger one)
@eliebakouch @AndrewCurran_ isn't each smaller run the proof for the next bigger one?