Researchers Release ScaleMRL Paper And Code For Multitask Reinforcement Learning

Why does it work? Our analyses suggest predictive objectives lead to: 💰Richer latent representations 💠Higher effective representation dimensionality 🧠Fewer dormant neurons ⚖️More stable value learning 8/🧵

2h192

LIKES2

Pablo Samuel Castro@pcastr

The broader takeaway is not that planning is unnecessary. Rather, we should carefully distinguish the benefits of representation learning from the benefits of planning. As RL continues to scale, understanding which ingredients matter is essential Paper and code are below. 👇

2h172

REPLIES1

Pablo Samuel Castro@pcastr

Paper: https://arxiv.org/abs/2606.05555 Code: https://github.com/johanobandoc/ScaleMRL Thanks for reading! If you're interested in multitask RL, representation learning, or scalable reinforcement learning, we'd love to hear your thoughts and feedback. I'll also be at #ICML2026 next week 🇰🇷

Pablo Samuel Castro@pcastr

The broader takeaway is not that planning is unnecessary. Rather, we should carefully distinguish the benefits of representation learning from the benefits of planning. As RL continues to scale, understanding which ingredients matter is essential Paper and code are below. 👇

2h1610

Pablo Samuel Castro@pcastr

What do those representations look like? Each color corresponds to a different task. Predictive representations lead to structured, high-dim latent spaces. W/out them, representations collapse & are less expressive. 9/🧵

2h192

Pablo Samuel Castro@pcastr

Perhaps even more interesting: The learned representations transfer remarkably well. MR.Q achieves stronger zero-shot performance on unseen tasks and adapts faster during few-shot finetuning, suggesting that better representations naturally lead to better transfer. 10/🧵

2h172

Pablo Samuel Castro@pcastr

The improvements go well beyond final performance. We observe: 📈Higher sample efficiency ⏱️Better wall-clock efficiency 🦣Better scaling with larger models 💪🏾Greater robustness in low-data regimes 🚀All without explicit planning. 7/🧵

2h172

Pablo Samuel Castro@pcastr

Modern world-model methods combine several ingredients: • Predictive objectives • Latent representations • Planning • Large neural networks We set out to determine which ones are actually responsible for the observed gains. 3/🧵

Pablo Samuel Castro@pcastr

What actually makes modern RL systems scale? Many recent successes attribute their performance to world models and planning. Our work suggests a different perspective: 👉 It may be representation learning, not planning, that's doing most of the heavy lifting. 2/🧵

2h610

Pablo Samuel Castro@pcastr

We eval on MMBench [1] (@ncklashansen), a benchmark w/ 200 diverse tasks spanning multiple domains and embodiments, each w/ lang instructions, demonstrations, and optional visual observations. We compare against Newt, a recent large-scale multitask world model built for it. 5/🧵

Pablo Samuel Castro@pcastr

If representation learning is key ingredient, then a purely model-free algo should achieve competitive multitask perf— even without planning. To test this, we revisit MR.Q (Fujimoto et al), a model-free RL algorithm and augment it with predictive auxiliary objectives. 4/🧵

2h210

Pablo Samuel Castro@pcastr

The results were surprisingly strong. MR.Q consistently matches or outperforms, the model-based baseline across diverse multitask domains, while relying on a significantly simpler learning pipeline and without explicit planning. 6/🧵

Pablo Samuel Castro@pcastr

We eval on MMBench [1] (@ncklashansen), a benchmark w/ 200 diverse tasks spanning multiple domains and embodiments, each w/ lang instructions, demonstrations, and optional visual observations. We compare against Newt, a recent large-scale multitask world model built for it. 5/🧵

2h210

Pablo Samuel Castro@pcastr

If representation learning is key ingredient, then a purely model-free algo should achieve competitive multitask perf— even without planning. To test this, we revisit MR.Q (Fujimoto et al), a model-free RL algorithm and augment it with predictive auxiliary objectives. 4/🧵

Pablo Samuel Castro@pcastr

Modern world-model methods combine several ingredients: • Predictive objectives • Latent representations • Planning • Large neural networks We set out to determine which ones are actually responsible for the observed gains. 3/🧵

2h300

Pablo Samuel Castro@pcastr

Finally, most of the credit goes to @johanobandoc who led this project! Thanks to a great set of co-authors @luli_airl, Scott Fujimoto, @pierrelux, and @AaronCourville for countless insightful discussions throughout this work.

Pablo Samuel Castro@pcastr

Paper: https://arxiv.org/abs/2606.05555 Code: https://github.com/johanobandoc/ScaleMRL Thanks for reading! If you're interested in multitask RL, representation learning, or scalable reinforcement learning, we'd love to hear your thoughts and feedback. I'll also be at #ICML2026 next week 🇰🇷

2h1710