Muennighoff Presents Prefix Sliding for Test-Time Scaling
Method drops less relevant intermediate tokens from LLM reasoning traces during inference.
Niklas Muennighoff posted about dropping most intermediate reasoning tokens in LLM traces. He noted that the prefix and recent tokens matter most, so the rest can often be removed. A retweet highlighted Prefix Sliding for efficient test-time scaling, where models discard tokens outside a recent window. The linked arXiv paper and GitHub repository from Muennighoff describe the approach as a way to reduce memory use when models reason longer on problems. No training changes are required.
Combined views
1.5K
2 posts, first seen 2d ago
Reactions
2 postsNM 
Niklas Muennighoff@Muennighoffwhen you read LLM reasoning traces, you may have wondered before how they are useful a few thousand tokens later they often aren’t -- the prefix (task, system instruction, tool definitions etc) & most recent tokens tend to matter most. can we just drop the rest?
- likes: 8
- replies: 1
- bookmarks: 4
- reposts: 0
TM Tanishq Mathew Abraham, Ph.D.@iScienceLuvrPrefix Sliding for efficient test-time scaling "we find most intermediate reasoning tokens lose importance as the model continues reasoning. This calls into question whether retaining them is worth the cost. Based on this insight, we propose Prefix Sliding, which discards…
- likes: 102
- replies: 10
- bookmarks: 67
- reposts: 16
