Math-data timing in LLM training appears to depend on the learning-rate schedule

The authors say their tests with fixed compute and mid-training token budgets found that mixing limited math data with web data earlier lowered mid-training loss when learning rates decayed throughout training. With a constant learning rate, saving mid-training data for the end worked better. Their best tested strategy kept the learning rate high after mid-training data entered, then decayed it near the end.