Weight Extrapolation of RL Checkpoints Produces Complementary Policies

🧵 For 2 RL checkpoints trained differently, you can just weight extrapolate them and it works! Bonus: these extrapolated checkpoints are complementary policies -> Get exploration and diversity for free -> Better inference scaling when ensembling Paper: https://arxiv.org/abs/2605.28751