Evaluation and optimizer choices in a toy continual-learning experiment

An experimenter reports that increasing SGD + PCA replay from two to four epochs—training passes—per task gets 91.1%. They report 95.2% for joint training with 10 passes through all tasks’ data together. They stress the limits: different training budgets, one toy benchmark and three seeds make this a useful reference, not a ceiling. Their takeaway is to check what evaluation may be hiding and treat loss scaling as part of the optimizer setup. They also share a research log described as containing the full experiment ladder, plots and code.