Learned Geometric Biases Enhance RL Task Optimization Despite Objective Deviations

an example of this is a task that takes paragraphs, sorts sentences into some ~random order, asks the lm to reconstruct a valid one there's vastly many more wrong orders than correct orders, so pinning to the original order is sane - but the original order isn't uniquely determined by the input in a lot of cases the value estimator, in principle, learns a way to assign relative points that hedge as best as is possible given the incompressible constraints. that's probably the true value (heh) in explicit estimation; not some vague """variance reduction""" for """sample efficiency""" in the sense that the canon literature likes to invoke (else robotics people could just crank GRPO group size), but the smooth manifold geometry with inherently relative characteristics

in general i just (sorry @hallerite) hate sample efficiency as a lens for this type of thing, to an almost prejudiced degree, because you're not getting any more or less "true bits" from learned value estimators, the only reason why a learned value estimator can do anything useful for you is because it learns structure in the geometric sense, much like a bottleneck representation the structure compounding nicely from a value head is, in some sense, equally as "wasteful" as GRPO informationally speaking, but forces it through a funnel that has invariant geometric characteristics i struggle to call that something that's more "sample efficient" in any meaningful sense, because it's not really a property of the reward function being """Learned Better""", but rather, the reward function being learned **to the extent that there is implied latent structure which is actually learnable** in the reward function