Ian Osband argued in an X thread that policy gradient can perform badly in image classification even in a setting where, he wrote, it is “exact” and avoids the exploration, credit-assignment and sampling-noise issues often blamed in reinforcement learning. In the thread, Osband cited his own reported ImageNet comparison of about 4% for policy gradient versus 62% for cross-entropy. (Osband on X)
Osband presented classification as a simplified setting for looking at the objective itself rather than those usual RL complications. The ImageNet figures here come from Osband’s thread and are reported as his comparison, not as an independently verified benchmark result. (Osband on X)
Why Osband says policy gradient misses low-probability wins
To explain the intuition, Osband compared two examples: one where the correct label already has probability 0.9, and another where the correct label has probability 1e-6. He wrote that for a tiny step, policy gradient treats improving the 1e-6 case as hardly worth it because the immediate success-rate gain is so small, even though over many steps it can be better to invest in that example than in the one that is already near-correct. (Osband on X)
In Osband’s framing, that is the short-horizon problem: the immediate improvement signal can make a low-probability correct answer look unimportant even when continued training would make it more valuable to improve. (Osband on X)
From “patient accuracy” to “horizon loss”
Osband described cross-entropy as “patient accuracy,” which he defined as “the total error an example would pay if training on it never stopped.” In the same thread, he described policy gradient as the zero-horizon limit. He then contrasted both with what he called “horizon loss,” which truncates that total by the remaining training budget. Osband wrote that this amounts to “one line of code.” (Osband on X)
He also wrote that the planning loss beats both “myopic policy gradient” and “fully patient cross-entropy.” In the same post, Osband said the same planning approach can apply to cross-entropy too, making this, in his telling, broader than a simple ranking between two standard losses. (Osband on X)
How he says the project got there
In a later reply, Osband said the project began as an attempt to explain why “delightful policy gradient” outperformed ordinary policy gradient. He wrote that the work kept simplifying the problem until it reached classification, and added that the usual “follow the gradient” story does not match what people do “even in this simplest deep learning setting.” (Osband on X)
Across the thread and reply, Osband’s posts frame the issue as one where a planning-style loss can capture value that a purely immediate update misses. That framing, along with the ImageNet comparison, comes from Osband’s posts rather than an independently confirmed report presented here. (Osband on X; Osband on X; Osband on X)
Comments