IsoVRBench uses synthetic video pairs to test video reward models

The IsoVRBench team says it produces preference labels without human annotation by changing one aspect of a video to make a lower-quality counterpart. It reports that video reward models struggle with shuffled or held frames. The team says its model, trained on 160,000 synthetic pairs, performed comparably to one trained on 180,000 human labels. It also reports better VBench-2.0 results and higher human preference when synthetic and human data were combined.