François Fleuret of Meta FAIR proposes a validation protocol requiring researchers to normalize FLOPs and memory before scaling from 1B to 200B parameters

you basically lose 3/4 of ideas at every step.

@francoisfleuret It's funny because almost everyone stops after the 3rd sometimes 4th step, writes a paper, submits it and then complains about the randomness of reviews.

@francoisfleuret Also normalize wall-clock time. Some flop savings are illusory

@francoisfleuret isn't it the standard workflow in any context?

@francoisfleuret who pays for the 32b and 200b runs? that part usually gets skipped

@roeeshenberg Yes absolutely.

@francoisfleuret why not going with 8b directly. 1b is just too small.

@giffmana @francoisfleuret the later steps do seem to involve dropping five or six figures on it

@letonyo Yes, but the pressure in DL/AI is intense.

@francoisfleuret Obviously

@__mbel__ @francoisfleuret iterating on 1b is nicer than 8b for the poors among us (me)

@francoisfleuret We need a platform that automatizes the steps >1B

@francoisfleuret I was just going to type that. Most ideas OOM at scale 1.

@francoisfleuret I had money up until the works at 32b part now what

@francoisfleuret but i only have my 3080 gpu to test so i can only do 1b

@giffmana @francoisfleuret pareto principle.

@sasuke___420 @francoisfleuret Really only the last two and i think it's fine to not have them in a paper

@francoisfleuret 3/4 loss per step means roughly 0.4% of ideas survive to 200b. The 1b to 8b transition is where most of mine go to die.

@francoisfleuret If an idea fails at 1b, how confident are you that it will fail at 200b?