RL is hot and has a reputation of being difficult to control. That is, unless you have the right parametrization. Off-policy RL, in particular struggles with the fact that your roll-out looks different from what you're actually optimizing. This can happen if you roll out with a more powerful model, or if you want to roll out in FP8, while you're training in FP16, or if you update asynchronously, or in other words, almost all the time in practice.
Users praised the P3O method for stabilizing off-policy RL training with FP8 rollouts, commending the authors for effectively mitigating common issues through targeted algorithms.
No Digg Deeper questions have been answered for this story yet.
#290The community has developed lots of algorithms to mitigate these problems, which work well, for a particular dataset, for a particular set of hyperparameters. Or you could simply quantify how close your policy is to what's being rolled out via effective sample size. This is what P3O does. Kudos to @rasoolfa Murdock Aubry and Nicholas Stranges from @boson_ai for pulling it off.
Blog: https://feynrl-project.github.io/blogs/episode_two.html Paper: https://arxiv.org/pdf/2605.12380 Code: https://github.com/FeynRL-project/FeynRL
RL is hot and has a reputation of being difficult to control. That is, unless you have the right parametrization. Off-policy RL, in particular struggles with the fact that your roll-out looks different from what you're actually optimizing. This can happen if you roll out with a more powerful model, or if you want to roll out in FP8, while you're training in FP16, or if you update asynchronously, or in other words, almost all the time in practice.
