The release includes complete training data, weights, and rollouts.
Trained some terminal agents with friends!
Introducing Tmax, open RL terminal agent models. Under default settings and shorter length (65k) token budgets, tmax outperforms prior open work on terminal use. We are releasing all data+weights+rollouts publically!
Many users praised the Tmax terminal agent models' public release of weights, rollouts, and sandbox for enabling tangible RL learning, while one questioned the benchmarking as potentially self-serving.
No Digg Deeper questions have been answered for this story yet.
Tmax consists of two parts: a dataset and a simple RL recipe. For our dataset, we constructed a synthetic pipeline where we: (1) sample values across 9 axes (2) prompt a strong generator model This allowed us to cheaply generate ~15k instances, which we publicly release!
Anyway, check out the paper for more details and results! I tried to release as much as I could (including rollouts during 9b training). Paper: https://github.com/hamishivi/tmax/blob/master/assets/paper.pdf Blog: https://wai-org.com/blog/tmax/ Models and data: https://huggingface.co/collections/allenai/tmax Code: https://github.com/hamishivi/tmax
We find that the RL training setup generalises extremely well: SWE-Bench Verified improves, AIME improves, and even performance in different harnesses improves!
The second part is RL: we design a simple RL setup, largely following DPPO, but using the asynchronous RL machinery from open-instruct. Training on top of Qwen 3.5 9B, we greatly improve performance and outperform other terminal-use datasets when training using our RL setup.
It's not all fun and games, though: we found the RL training quite hard to stabilize, and we document a few things we tried (e.g., DPPO, larger group sizes). I think there's still a lot more to do here (PPO?), and having longer, stabler runs likely leads to improved performance.
This was joint work with the awesome @yinn_oscar and lots of thanks to my superstar collaborators @RulinShao @TengX6 @natolambert @HannaHajishirzi!
@hamishivi @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi @waiorg If people could use this via our harness would you be interested? I'm building an opensource harness and was thinking about providing such models and if people use, you get paid but ofc the initial setup would cost you
They first define a set of tasks which serve as seed values. Each of these 9 axes are sampled and composed together to be fed to a task generator
First, a good pipeline should be: 1) Scalable. It should not be too challenging infrastructure-wise, given the sheer scale of data required 2) Some form of diversity 3) Calibrated difficulty for models (neither too high or too low pass rates)
@hamishivi @RulinShao @yinn_oscar @TengX6 @natolambert @HannaHajishirzi @waiorg Would you say that open source + fine-tuning for a specific domain is more promising than SOTA closed models in terms of cost efficiency?
I think it depends on how far out what you want to do is from the current ‘hot thing’ coding agents are fairly popular so unlikely you can really push it. But for things like long form outputs (like harveys law agent stuff or my past DR Tulu work) you can definitely get cheaper higher quality outputs by finetuning (albeit with some higher upfront cost).
@YashasGunderia @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi @waiorg potentially, although I see this as more of an academic artefact! I think these models are likely still a bit tough to use in practice (mostly due to their small size).
Training is pretty standard: - GRPO + mismatch masking - FP32 lm head. This is worse for Qwen 3.5 likely due to its hybrid nature - Max length 65K, 100 steps, group size 32, batch size 256
@hamishivi @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi @waiorg Respect for actually shipping the sandbox layer! Your ceiling is container start, the docker daemon caps concurrent creates (the 64 cap + janitor). Fork off a warm snapshot and it's gone: sub-ms, no daemon. Built Collimate for this, happy to share an open-instruct backend!
Because data is sampled via the 9 axes, they can explicitly control for diversity/balance with considerable difficulty
The whole pipeline: - Skips difficulty calibration using the teacher model (which can get expensive) and instead performs online filtering - Diversity is achieved from the sampling above + augmented with personalities - Difficulty is calibrated from the complexity axes
This results in significantly stronger Terminal Bench scores. They also show that their setup has the highest number of steps which is a decent proxy for difficulty
@hamishivi @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi Btw why without online rl?
@justALEXWORTEGA @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi We train with online RL! having a good online setup was a chunk of the work for this :)
@hamishivi @yinn_oscar @RulinShao @TengX6 @natolambert @HannaHajishirzi @waiorg Same read. The details give it away.
@hamishivi Thank you for sharing 🙏 awesome work. I'm teaching myself RL and having something this tangible to work with really helps.
