RL batch sizes consistently equaled or exceeded pretraining sizes.
In general RL seems to have a very different batch size scaling than pretraining Also even in their previous shorter context stages, the batch sizes were always bigger or equal to the pretraining batch size
Interestingly didn’t see anyone talking abt this but MAI used a batch size of almost 1B tokens during their final RL stage
@Grad62304977 @yacineMTB I’d guess this is a variance problem characteristic to a good bit of RL in general?
There is a wealth of variance reduction literature, control variants, approximate/surogate gradients through a learned model of environment, that I’ve been waiting to see if it makes its way into modern RL, like REBAR, RELAX, “backpropagation through the void”
RL batch sizes consistently equaled or exceeded pretraining sizes.
In general RL seems to have a very different batch size scaling than pretraining Also even in their previous shorter context stages, the batch sizes were always bigger or equal to the pretraining batch size
Interestingly didn’t see anyone talking abt this but MAI used a batch size of almost 1B tokens during their final RL stage
@Grad62304977 for training stability I guess
@Grad62304977 @yacineMTB I’d guess this is a variance problem characteristic to a good bit of RL in general?
There is a wealth of variance reduction literature, control variants, approximate/surogate gradients through a learned model of environment, that I’ve been waiting to see if it makes its way into modern RL, like REBAR, RELAX, “backpropagation through the void”