Research finds LLMs represent goal success probability as an internal linear axis similar to reinforcement learning

Related to recent work by @andy_q_han!

New work: The Value Axis 🎯

How do LLMs choose which path to take mid-task? We find they internally track the chance of reaching their goal along a linear axis, akin to a value function in RL. We show it modulates confidence in math & coding and can be reshaped with DPO and SFT.

Finally, we apply the value axis to in-the-wild settings. For instance, we find that after post-training, the highest-confidence Chatbot Arena queries are information-extraction tasks, whereas the least-confidence are politically sensitive queries-e.g "Is Taiwan part of China?"

Our work suggests that language models use a general, internal notion of “likely to do a good job” to decide whether to change direction or stay the course. More details / case studies (e.g effects of SFT, eval awareness) in our paper: https://arxiv.org/abs/2606.17056

Method: we synthesize "in-context RL" conversations where the model (Qwen3-8B) guesses a hidden criterion for editing a paragraph and gets +1/−1 feedback. To make the axis, we contrast tokens just before vs. after it uses the criterion, the point where the value should jump.

We find that the value axis tracks the model’s confidence in its current reasoning trajectory across math and coding tasks. Activations distinguish whether the model thinks its answer is correct, whether it backtracks, and whether code is correct.

The internal value function can be reshaped during post-training. Using DPO to make models prefer a specific word, we find that internal value increases on that word in the assistant response after training, suggesting models become more confident after doing rewarded behaviors.

We can also change the model’s confidence (e.g in coding). Steering towards high value increases verbalized confidence on AIME questions and reduces the amount of justification given for coding problems. Steering towards low value increases backtracking.

In fact, when instructed to use these preferred words (e.g for naming variables), models spuriously produce fewer comments, type hints, and lines of code after DPO, consistent with the prior value-steering effects. The effect is reversed if we train models to avoid the words.

This project was a wonderful collaboration with @ikauvar and supervised by @Jack_W_Lindsey as part of Anthropic Fellows!

Markdown version: https://github.com/nickjiang2378/value-axis/blob/main/value_axis.md

Code: https://github.com/nickjiang2378/value-axis

@nickhjiang Does this work for detecting (or even removing) sandbagging?

@nickhjiang