Cody Blakeney recommends maintaining a fixed parameter-to-token ratio.
I'm training a big model w/ P parameters and D tokens. Before I do so, I train smaller models to estimate the performance of the big model. Which of the following scaling regimes should I use to get the best predictions? 1. Fix D across all models 2. Fix P/D 3. Something else
No Digg Deeper questions have been answered for this story yet.
I expect the answer to be #2, but I haven't been able to find any evidence for it.
GPT-5.5 says #1 Opus 4.8 says #2 for "understanding scaling" but #1 for "predicting the final loss" Fable says #2
I'm training a big model w/ P parameters and D tokens. Before I do so, I train smaller models to estimate the performance of the big model. Which of the following scaling regimes should I use to get the best predictions? 1. Fix D across all models 2. Fix P/D 3. Something else
@BlancheMinerva The one caveat I would throw in here is that you should downsample your unique tokens to simulate the epoching that will be done at the full scale of P and D.
@BlancheMinerva If you already know P and D, then do 2.
If you are trying to decide P and D, then it's slightly more complicated.
@BlancheMinerva If you already know P and D, then do 2.
If you are trying to decide P and D, then it's slightly more complicated.
I'm training a big model w/ P parameters and D tokens. Before I do so, I train smaller models to estimate the performance of the big model. Which of the following scaling regimes should I use to get the best predictions? 1. Fix D across all models 2. Fix P/D 3. Something else
predict the benchmark performance of my model on downstream tasks or the behavior of my model in other ways like “which sequences will my model memorize.” The question I’m trying to ask is “how should I get the most accurate prediction of the final benchmark values.”
Most people seem to have interpreted this as “how do I find the optimal scaling regime.” That’s not what I’m asking. Let’s say that I’ve determined the compute-optimal frontier and have decided what P and D I want to use for my large training. Now, after doing that, I want to…
Most people seem to have interpreted this as “how do I find the optimal scaling regime.” That’s not what I’m asking. Let’s say that I’ve determined the compute-optimal frontier and have decided what P and D I want to use for my large training. Now, after doing that, I want to…
I'm training a big model w/ P parameters and D tokens. Before I do so, I train smaller models to estimate the performance of the big model. Which of the following scaling regimes should I use to get the best predictions? 1. Fix D across all models 2. Fix P/D 3. Something else
@code_star
Most people seem to have interpreted this as “how do I find the optimal scaling regime.” That’s not what I’m asking. Let’s say that I’ve determined the compute-optimal frontier and have decided what P and D I want to use for my large training. Now, after doing that, I want to…