‘Forking’: A reported sudden overfitting pattern under data replay

A paper on arXiv describes “forking” as a sudden divergence between training and validation loss when models with over-encoding n-gram memory repeatedly train on the same data. Its authors say they reproduced the pattern in a DeepSeek-style Engram model and found that low-frequency contexts contributed most of the gap. Co-author Ziming Liu says freezing the n-gram table helps during post-training.