It merges multiple RL experts offline using only checkpoints.
Users appreciate the isospectral optimization stack because it reuses the base model spectrum to cut RL training steps by 2.7x.
No Digg Deeper questions have been answered for this story yet.
Very insightful work! RL barely change the weights and mostly rotate the weight in spectral space, therefore opitmizing in the subspace will bring a better training effiency.
People keep asking me: what's different about optimization in RL? Seemingly nothing — the pre-training stack just works (Adam, even SGD 👀 @saagnikkk). Bringing some answers from my last work (sorry for the delay — been cooking 🚀). We introduce ISO: Isospectral Optimization: an RLVR-native optimization stack. Built on one simple observation, spectral inheritance: RLVR can reuse the base model's spectrum and acquire new behavior purely through the singular frames. 🧩 Offline: ISO-Merger — consolidates RL experts into one model with no data, no rollouts, no OPD. Checkpoints only. ⚙️ Online: ISO-Optimizer — a drop-in wrapper on AdamW / Muon that matches AdamW's accuracy with ~2.7× fewer steps on Qwen3-8B-Base. 📄 https://arxiv.org/abs/2607.19331 🌐 https://iso-rlvr.github.io/ 🧵👇
People keep asking me: what's different about optimization in RL? Seemingly nothing — the pre-training stack just works (Adam, even SGD 👀 @saagnikkk). Bringing some answers from my last work (sorry for the delay — been cooking 🚀). We introduce ISO: Isospectral Optimization: an RLVR-native optimization stack. Built on one simple observation, spectral inheritance: RLVR can reuse the base model's spectrum and acquire new behavior purely through the singular frames. 🧩 Offline: ISO-Merger — consolidates RL experts into one model with no data, no rollouts, no OPD. Checkpoints only. ⚙️ Online: ISO-Optimizer — a drop-in wrapper on AdamW / Muon that matches AdamW's accuracy with ~2.7× fewer steps on Qwen3-8B-Base. 📄 https://arxiv.org/abs/2607.19331 🌐 https://iso-rlvr.github.io/ 🧵👇
[5/8] ⚙️ ISO-Optimizer: run your optimizer in fixed-spectrum coordinates. It parameterizes W = U·Σ₀·Vᵀ, freezes Σ₀ from the base model, and lets AdamW / Muon act on (U, V), with a polar retraction to stay feasible. No new update rule — just the right coordinates. Results (Qwen3-8B-Base, math RLVR): AdamW: 0.495 after 270 steps — then plateaus (we ran it longer to rule out under-training). ISO-AdamW: 0.495 by step 100 → 0.509 at step 210. Same pattern at 1.7B / 4B (~2.2× fewer steps) and with Muon (~1.4×). And the cost? The polar retraction adds only ~7% to end-to-end step time at 4B scale.
[2/8] The observation, concretely: RLVR barely moves the singular values of your weight matrices. Spectral drift ≈ 10⁻²% of the weight scale, explaining only ~3% of the total update — at the endpoint AND along the whole trajectory. SFT, by contrast, rewrites the spectrum (~35%). What moves: the singular frames (U, V) — each layer's input/output directions.
[3/8] Is the spectrum functionally reusable? We tested it three ways: • Restore the base spectrum after RL (keep the learned frames) → gains preserved • Freeze the base spectrum for the entire run → still learns strongly (here, even beats AdamW) • Train only the spectrum, freeze the frames → barely learns That's spectral inheritance, verified: the spectrum is reusable and the frames do the learning.
@saagnikkk [4/8] So the recipe writes itself: inherit the spectrum, optimize the frames. One subtlety we verified first: BOTH frames must stay adaptable — freezing either side leaves ~half the checkpoint update unexplained; fixing only the spectrum leaves ~2%.
[7/8] So — what's different about optimization in RL? Pre-training decides the spectrum. RLVR learns how it acts. Inherit the spectrum, optimize the frames. 📄 Paper: https://arxiv.org/abs/2607.19331 💻 Code: https://github.com/zhuhanqing/ISO 🌐 Site: https://iso-rlvr.github.io Builds on our earlier analysis of RL's optimization dynamics, "The Path Not Taken": https://arxiv.org/abs/2511.08567
[6/8] Offline, the same coordinates solve the problem OPD is usually hired for: consolidating RL specialists into one model. ISO-Merger needs only the checkpoints — no post-merge data, no rollouts, no gradient updates, no distillation. It recovers ~99.5–106% of each specialist's score and posts the strongest aggregate among data-free merging methods (Task Arithmetic, TIES, TSV, RAM, OrthoMerge-G-TIES) on both testbeds.
@saagnikkk [8/8] ISO wouldn't exist without this team ❤️ @CongWenyan0320 — my closest friend. @ZhizhouSha, @saagnikkk, Xinyuan Song, David González-Martínez, @XiaoxiaWShirley, @tydsh, @Shiwei_Liu66, David Z. Pan, and Atlas Wang.
@jiayiy @saagnikkk Need Jiayi in the optimization loop 🫣
@zhu_hanqin41424 what's different about optimization in RL?
@jiayiy @zhu_hanqin41424 @saagnikkk signal to noise ratio
@zhu_hanqin41424 @saagnikkk what's different about optimization in RL?