Positive users praise the LLM-as-a-verifier method for making self-improvement loops more practical in agentic systems, while a negative user calls the Terminal-Bench results presentation misleading.
Based on 5 visible X reactions from 6 accounts; directional sample.
Ask a question below.
Published answers will appear here.
I think this is a misleading way to present the Terminal-Bench results. The harness deliberately runs each trial in one isolated sandbox whose end state gets graded. Running 5 sandboxes per task and having a judge pick the best graded run is a different game, and there's no guarantee it's even the best use of 5 sandboxes' compute.
@Azaliamirh This is exactly where agents need to go: not just generating answers, but scoring, comparing, and selecting better actions before committing.
@Azaliamirh This is clean and practical — love how you're making self-improvement loops more accessible for agentic systems.
@Azaliamirh So much alpha! Best follow yet!!
The method uses 1-to-20 scoring scales instead of binary rewards.
I think this is a misleading way to present the Terminal-Bench results. The harness deliberately runs each trial in one isolated sandbox whose end state gets graded. Running 5 sandboxes per task and having a judge pick the best graded run is a different game, and there's no guarantee it's even the best use of 5 sandboxes' compute.
@Azaliamirh +1
Check out LLM-as-a-Verifier: a simple, cheap, & general-purpose self-improvement technique that boosts performance on "any" agentic task we've tried. It achieves SOTA on Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench. The key idea: - Use fine-grained scoring granularity (e.g. 1-20) - Scale model responses with repeated sampling and criteria-based scoring - Rank results based on the expected logprobs of said scores We made it easy for you to try: Code: https://github.com/llm-as-a-verifier/llm-as-a-verifier Claude Code Plugin: https://github.com/llm-as-a-verifier/TurboAgent Paper: https://arxiv.org/pdf/2607.05391 Work is led by @jackyk02, with an awesome team!
Positive users praise the LLM-as-a-verifier method for making self-improvement loops more practical in agentic systems, while a negative user calls the Terminal-Bench results presentation misleading.
Based on 5 visible X reactions from 6 accounts; directional sample.
Ask a question below.
Published answers will appear here.
Check out LLM-as-a-Verifier: a simple, cheap, & general-purpose self-improvement technique that boosts performance on "any" agentic task we've tried. It achieves SOTA on Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench. The key idea: - Use fine-grained scoring granularity (e.g. 1-20) - Scale model responses with repeated sampling and criteria-based scoring - Rank results based on the expected logprobs of said scores We made it easy for you to try: Code: https://github.com/llm-as-a-verifier/llm-as-a-verifier Claude Code Plugin: https://github.com/llm-as-a-verifier/TurboAgent Paper: https://arxiv.org/pdf/2607.05391 Work is led by @jackyk02, with an awesome team!