CMU Researchers Introduce Gym-Anything to Turn Any Software Into AI Agent Environments

How Gym-Anything turns a new software app into a verified agent environment.

One creation agent sets up the software and collects evidence like screenshots and logs, while a separate audit agent checks whether the setup is good enough and sends feedback if it is not.

They start from real job and GDP data, map that to thousands of software tools, filter for apps that can run in a test sandbox, then keep a balanced set across important work areas.

The big deal is that their benchmark is tied to real economic work, so the agent tasks are meant to reflect software people actually use, not just easy demo apps.

This figure shows the full Gym-Anything pipeline: they pick important real-world software, turn each app into an environment where an AI agent can act, then create many realistic tasks inside those apps.

The big deal is that benchmark creation becomes much less manual, because one agent builds and another checks the environment before other agents are tested on long software tasks.

It also shows why the result matters: when agents are tested this way, the tasks look more like real office work, and current agents still struggle.

@rohanpaul_ai real work is the filter, toy tasks are too kind. this kind of benchmark feels closer to the truth.

@rohanpaul_ai the data distribution problem is brutal here. agents trained on one app's interaction patterns often can't generalize even to a slightly different UI for the same task. wonder if CMU's approach does anything about that or just expands the surface area of the same problem.

@rohanpaul_ai The 'agents can set up their own training env' part is honestly the more interesting angle — if the benchmark can bootstrap itself, the eval loop scales with compute instead of grad student hours.

@rohanpaul_ai One agent building the environment while another audits the proof is a smart way to scale realistic benchmarks.

@rohanpaul_ai any software becoming a training ground is the easy half. judging long messy work across totally different apps is the hard part, and that judge is the thing you actually build and maintain. the environment is free, the eval never is

@rohanpaul_ai yeah the security implications are definitely wild

@rohanpaul_ai This is the benchmark gap that matters: real software work is not one clean task, it is setup, state, recovery, and proof that the action actually changed the right thing. I'd love to see agents scored on repair loops as much as final success.

@rohanpaul_ai A 2B distilled model outperforming 2x larger ones says a lot. Training on realistic environments matters more than raw scale. Quality of training distribution > parameter count.

@rohanpaul_ai The hard part is the boring tail: app state, permissions, undo paths, and logs. Benchmarks that include those failure modes would be a lot more useful than another tidy task list.

Tying the benchmark to actual economic output instead of abstract tasks is the missing piece most agent evaluations skip. Most "benchmarks" measure how well an agent can solve contrived puzzles — this measures whether it can do work that actually moves GDP.

The real test will be cost per unit of economic output. If an agent can process 10x the volume at 1/10th the cost of a human, even imperfect accuracy becomes viable. That's where we'll see adoption curve inflection — not when agents pass some arbitrary score, but when their marginal cost falls below the marginal value they create.

CMU framing this as "real job and GDP data" rather than "can it write code" is exactly right.