Eric Jang argues AlphaGo's Monte Carlo Tree Search bypasses the credit assignment problems plaguing naive LLM reinforcement learning

Monte Carlo Tree Search training corrects the model move by move, while current LLM training only tells it whether the whole trajectory worked.

MCTS is preferable if you can get it. But nobody's managed to get MCTS to work for language models.

In his blackboard lecture @ericjang11 talked to me about why:

I can't wait for codex to be able to do all its work unsupervised so I have the time to watch all the @dwarkesh_sp 's blackboard lectures.

Watch Eric's full lecture here: https://www.dwarkesh.com/p/eric-jang

Watch the full episode here: https://www.dwarkesh.com/p/eric-jang

@dwarkesh_sp Feels like the same gap we hit with agentic coding: you need step-level verifiers, not just end-to-end reward. Without a cheap partial-state scorer, MCTS stays mostly theoretical for language.

The MCTS-for-LLMs ceiling is really about rollout cost, not search itself — every leaf evaluation is a full model call worth dollars and seconds, so the thousands of sims Go used per move are economically out of reach. Process reward models are the practical shadow of move-by-move credit until rollouts get an order of magnitude cheaper.

@DnuLkjkjh @dwarkesh_sp there's been efforts and it still reward hacks that step level reward by early exiting resulting in poor reasoning accuracy where it doesn't go deep enough to reason about the problem since it's too scared because of step penalty

@dwarkesh_sp step-level credit assignment is the part i’d want in coding agents too. outcome-only training hides whether the miss was tool choice, edit order, or bad context packing.

@dwarkesh_sp @grok can you summarize and provide resources for building with this?

@dwarkesh_sp These clip intros make each piece so easy to learn. Almost in a turn-your-brain-off sort of way. They spike my curiosity and prime me to get into the concepts without the friction of missing a piece or two.

Not always easy to do with STEM

Exactly. Outcome-only supervision works okay for chat, but it’s a nightmare for agents that have to operate step-by-step in noisy or physical environments.

If you can’t tell the model which specific action was wrong (vs just “the whole plan failed”), learning robust recovery and guardrails becomes extremely sample-inefficient.

@dwarkesh_sp MCTS works because it turns trajectory-level failure into move-level correction.

the missing piece is a verifier that can score partial semantic states without collapsing into final-answer reward.

Once partial states become auditable, search trajectories can be retained.

@dwarkesh_sp This is for deterministic environment with perfect information. For incomplete information or stochastic environment, you need CFR

@dwarkesh_sp Loved the episode, please do more in this format

@dwarkesh_sp Dwarkesh I think you’re the only hope to fix the literacy crisis post AGI

@dwarkesh_sp Move-level feedback is the missing granularity. Whole-trajectory reward tells an agent that something went wrong; it does not teach where the plan bent. Long-horizon agents need credit assignment you can inspect step by step.

@dwarkesh_sp "Current LLM training only tells it if the whole trajectory worked" No man, even the academia today has moved way beyond R1. On policy self correction like techniques are ubiquitous in academic articles.

@dwarkesh_sp For agents, PUCT needs a cost term too. The branch with the best expected value may also be a 40-tool-call rabbit hole. Planning policy should score reward, uncertainty, wall-clock, token burn, and reversibility before expanding.

@dwarkesh_sp the honest answer is that scaling mcts to language models requires a fundamental shift in how we think about optimization and feedback loops. current approaches just aren't designed to handle the complexity