AI Judge changed title after evaluation, original title: "Terminal-Bench Science extends its platform, already used by Anthropic, OpenAI, and Google DeepMind, with an open call for over 100 scientific workflow tasks for AI agent evaluation"
Contributors package workflows as RL environments with verification tests.
Ludwig Schmidt#440
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
Many users welcomed Terminal-Bench Science for creating meaningful benchmarks that bring AI agents closer to real scientific workflows, while others dismissed the effort as vanity metrics or premature automation before labs have solid data.
No Digg Deeper questions have been answered for this story yet.
I'm very excited about this extension to the celebrated Terminal-Bench to science.
If you're a scientist (life, physical, earth, mathematical science, etc) interested in AI, definitely check this out!
Terminal bench evaluate how good AI models are at controling tools on a computer to achieve a goal (using the command line). T-Bench science now extends that to "AI for Science" and it comes with a call to contribute your own (real scientific world) workflow to the benchmark (until August 2026).
The more workflows and the more diverse they are, the better the next generation of AI models will be at helping you in your daily research work.
Note that this is not a training dataset, it's to evaluate frontier model performances.
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
let the hill climbing on scientific tasks begin
new benchmark: TerminalBench Science
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
Terminal-Bench Science is a direct way to contribute to AI for Science. It's programming agents by task specification. Ask a precise scientific question and watch how AI agents will learn to solve it: Step 1. Package a scientific task or workflow, something that takes a working scientist a week or month to do into an RL environment. Step 2. Write tests that verify if the task has been done correctly (can be done easily if you have already solved the task manually). Step 3. Sit back and let AI agent progress solve it in 6 months.
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
"AI for science" benchmarks today mostly test textbook recall. Terminal-Bench Science is a chance for scientists to practice writing that definition. Contribute a real workflow, and you find out exactly where today's best agents break on it. http://tbench.ai/news/tb-science-announcement
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
Hosted by @Stanford @StanfordAILab @StanfordHAI, @LaudeInstitute and @harborframework.
With @lschmidt3, @sanmikoyejo, @AlexGDimakis, @bradenjhancock, @JJitsev, @ryanmart3n, @alexgshaw, @Mike_A_Merrill, @LinShi592021, @Krauth, @stebo85, @0xrobertzhang, @rishi_desai2, @ekellbuch, @xdotli, @realjustinbauer, @HeckelReinhard, @oq_35, @YuanqiD, @chenru_duan, @hcwww_, @gregd_nlp, @russpoldrack, @RisaWechsler, @SnorkelAI and a growing community of contributors and advisors across the natural sciences.
6/6
⏰ Deadline: August 17, 2026 — the earlier you start, the more time we have to help your task land.
📋 Submit a task proposal: https://airtable.com/appzZC5gEHrXSfNNw/pagjgS95lAQ5FVJxt/form 💻 GitHub: https://github.com/harbor-framework/terminal-bench-science 💬 Discord (tb-science): https://discord.gg/ZvcWupVXjz 📅 Weekly Meeting (Mondays, 9am PT): https://meet.google.com/yco-yhwc-sid 📩 Contact: stevendi@stanford.edu
5/6
Why contribute:
🎯 Make AI better at your science. Your tasks set what frontier labs optimize for. 🛠️ Gain agentic eval experience. See where today's best AI agents succeed and fail. 📝 Become a co-author. Every merged task earns authorship on the Terminal-Bench Science paper.
3/6
Terminal-Bench Science is built by the scientific community to shape AI for science.
Most "AI for Science" benchmarks test textbook knowledge or contrived toy problems. We measure real computational workflows scientists run in practice.
Got a complex scientific workflow you wish an AI agent could handle? We want it.
2/6
Extremely excited for Terminal-Bench Science, which we're proud to support via our Open Benchmarks Grants @SnorkelAI !
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
Terminal-Bench Science is built by the scientific community to shape AI for science.
Most "AI for Science" benchmarks test textbook knowledge or contrived toy problems. We measure real computational workflows scientists run in practice.
Got a complex scientific workflow you wish an AI agent could handle? We want it.
2/6
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
What a task looks like — real workflows from scientific domain experts:
- Reconstructing MRI brain maps - Virtual drug screening - Reconstructing ice crystal disorder over time
Scientifically grounded, programmatically verifiable, hard for today's best AI agents. Browse the full set on GitHub: https://github.com/harbor-framework/terminal-bench-science/tree/main/tasks
4/6
Why contribute:
🎯 Make AI better at your science. Your tasks set what frontier labs optimize for. 🛠️ Gain agentic eval experience. See where today's best AI agents succeed and fail. 📝 Become a co-author. Every merged task earns authorship on the Terminal-Bench Science paper.
3/6
If you know a scientist whose workflow belongs in Terminal-Bench Science — RT this thread or send it their way 🙏
Hosted by @Stanford @StanfordAILab @StanfordHAI, @LaudeInstitute and @harborframework.
With @lschmidt3, @sanmikoyejo, @AlexGDimakis, @bradenjhancock, @JJitsev, @ryanmart3n, @alexgshaw, @Mike_A_Merrill, @LinShi592021, @Krauth, @stebo85, @0xrobertzhang, @rishi_desai2, @ekellbuch, @xdotli, @realjustinbauer, @HeckelReinhard, @oq_35, @YuanqiD, @chenru_duan, @hcwww_, @gregd_nlp, @russpoldrack, @RisaWechsler, @SnorkelAI and a growing community of contributors and advisors across the natural sciences.
6/6
Science is the frontier of AI. Contribute to this initiative if you can!
📣 Announcing Terminal-Bench Science: benchmarking AI agents on real scientific workflows – now open for task contributions👇
http://tbench.ai/news/tb-science-announcement
@AnthropicAI, @OpenAI, and @GoogleDeepMind use Terminal-Bench to evaluate AI on coding tasks. We're now extending it to scientific workflows.
1/6🧵
it's currently still being built and you can submit your verifiable tasks until august 17th 2026
let the hill climbing on scientific tasks begin
new benchmark: TerminalBench Science
⏰ Deadline: August 17, 2026 — the earlier you start, the more time we have to help your task land.
📋 Submit a task proposal: https://airtable.com/appzZC5gEHrXSfNNw/pagjgS95lAQ5FVJxt/form 💻 GitHub: https://github.com/harbor-framework/terminal-bench-science 💬 Discord (tb-science): https://discord.gg/ZvcWupVXjz 📅 Weekly Meeting (Mondays, 9am PT): https://meet.google.com/yco-yhwc-sid 📩 Contact: stevendi@stanford.edu
5/6
@StevenDillmann @BlancheMinerva @AnthropicAI @OpenAI @GoogleDeepMind I literally put in my notes app this morning to look into science agents haha will dig in
@evijit @BlancheMinerva @AnthropicAI @OpenAI @GoogleDeepMind let's chat! @evijit
@StevenDillmann @AnthropicAI @OpenAI @GoogleDeepMind I posted this on a colleague of yours, but i was curious about the scope of the workflow tasks. I could see something like this being valuable for psychopy and jsPsych experiment creation.
@StevenDillmann @AnthropicAI @OpenAI @GoogleDeepMind This is a great step for scientific research.
@StevenDillmann @AnthropicAI @OpenAI @GoogleDeepMind A meaningful bridge to real science benchmarks.
@StevenDillmann @BlancheMinerva @AnthropicAI @OpenAI @GoogleDeepMind Let’s DM! 🫡
