Flawed Evaluation Masks True Benefits of Process Rewards in AI Training

CRCameron R. Wolfe, Ph.D.@cwolferesearch

A lot of research has dismissed the benefits of process rewards over the last few years, but the way that we test if process rewards are helpful is oftentimes flawed IMO. If we are testing the benefit of process rewards versus pure outcome rewards, we need to be careful with how we perform evaluation. In particular, we should not use the outcome reward / final accuracy as the primary evaluation metric. If we do this, then of course training with pure outcome rewards will perform similarly to or better than outcome + process rewards. Training with pure outcome rewards directly optimizes the main metric we are using for evaluation. Process rewards will play a massive role in the future of AI. However, the benefit of process rewards may not be obvious if we are only looking at accuracy. It is very possible that outcome rewards provide more than enough signal to optimize an LLM / agent's accuracy. Even if this is the case, process rewards will help to optimize how we reach a correct final solution, which is oftentimes equally important to the correctness of the final solution. These are two equally important dimensions of model quality. As a concrete example, we could train a coding agent using pure outcome rewards and achieve good accuracy. However, we may also integrate a variety of process rewards that check the style, structure, and cleanliness of the code. Maybe these process rewards are unnecessary to achieve an accurate final solution. But, they are extremely beneficial in practice because they produce a coding agent that writes code that is both elegant and accurate (instead of just accurate). Some of these points might be obvious, as I think process rewards are already heavily used in many production RL settings. However, I still think taking a deeper look at this research area provides a nice example of how the way we evaluate techniques may heavily influence the findings that we get (and in turn change the trajectory of research!).