Freeform Preference Learning Improves AI Supervision With Multi-Axis Feedback

Project led by @marceltornev, @anubhamahajan01, @AbhijnyaBhat

Paper: https://arxiv.org/abs/2606.32027 Code & videos: https://freeform-pl.github.io/fpl.website/

Check out Marcel’s thread for more details!

Freeform preference learning has multiple nice properties:

(1) It works better

When controlling for the number of preference queries, learning with multi-axis preferences yields far more performant policies than single-axis rewards.

(3) Long-horizon credit assignment Most robot RL focuses on short horizon tasks b/c dense temporal rewards are hard to get.

Freeform preferences yield dense rewards for subtasks without subtask segmentation.

With freeform preference data, we train a lang-conditioned reward that captures all axes of a task.

We then train a policy conditioned on each reward axis and the corresponding reward.

FPL allows the robot to maximally leverage and learn from each axis of supervision.

(2) Compositional generalization By supervising with multiple axes, the robot can learn behavior that’s not in the data, e.g. fast behavior for a task that has only slow episodes in the data.

This isn’t possible with traditional reward models.

@chelseabfinn @marceltornev @anubhamahajan01 @AbhijnyaBhat Can the VLM be trained to output pareto-optimal rewards? If I understand this work: a human in the loop defines criteria in natural language, VLM scores video on the criteria, and the VLM & policy gets continually trained on an evolving loss fn as the criteria dimensions change.

@chelseabfinn @marceltornev @anubhamahajan01 @AbhijnyaBhat Multi-axis preferences making rewards actually interpretable, love this. Binary comparisons always felt like throwing away signal. 38% improvement is no joke either.