GRPO may not work as well with human-provided samples as with on-policy generated ones

A commenter reasons that rewriting training data to optimize for surprise only where it matters gives backpropagation a cleaner signal. They suggest GRPO might not work as well with human-provided samples instead of on-policy generated ones.