Shared-prefix trees reportedly sped up RL forward and backward passes by over 26x in experiments
The poster says long agentic rollouts make forward and backward passes a major bottleneck at large batch sizes.
TLDR
A poster says asynchronous reinforcement learning works best with large batch sizes, but forward and backward passes become a major bottleneck, especially when agentic rollouts exceed 100,000 tokens. In their experiments, a tree structure that shares prefixes across trajectories sped those passes up by more than 26 times. They credit Amine with the improvement to TRL.
Combined views
—
1 Source, first seen ago
— likes— comments— saves— reposts
