Builder Expresses Bearishness on N=1 RL for Length Penalties
Expanding batch size to compensate also reduces training data efficiency.
In a post on X, @willccbb argued they are "kinda bearish" on N=1 reinforcement learning for length penalties, saying the setup tries to reward efficiency without opening "hack backdoors for early-exit" while also offering no advance way to know how many tokens a task actually needs. In a follow-up reply on X, the same account also questioned whether a single forward pass should be trusted to estimate advantage in systems with "agentic judges."
Combined views
4.4K
5 posts, first seen 1h ago
Builder Expresses Bearishness on N=1 RL for Length Penalties
Expanding batch size to compensate also reduces training data efficiency.
In a post on X, @willccbb argued they are "kinda bearish" on N=1 reinforcement learning for length penalties, saying the setup tries to reward efficiency without opening "hack backdoors for early-exit" while also offering no advance way to know how many tokens a task actually needs. In a follow-up , the same account also questioned whether a single forward pass should be trusted to estimate advantage in systems with "agentic judges."