XLang NLP releases OSWorld 2.0 to test computer-use AI agents on complex, multi-hour workflows

VIEWS2.7K

🧵[2/8] What does "long-horizon and real-world" mean here?

In one reimbursement task example, the agent must: 📄 follow a tutorial PDF 🏢 operate a legacy ExpenseFlow portal 📬 extract amounts from noisy receipt emails 🏦 cross-check evidence across bank transactions and email 📨 notice a new email that changes the task mid-execution 🔎 recover hidden employee info from a prior report 🧾 prepare valid supporting documents ⚠️ detect inconsistent evidence 🙋 ask the user instead of guessing ✅ complete final review and submission

This is the kind of workflow where current agents still break.

3d2.7K291

BOOKMARKS3

XLANG NLP Lab@XLangNLP

🧵[6/8] The failures are the main lesson.

Agents execute local actions well, but fail to maintain task-level state over long horizons:

🧭 stated constraints drift after many steps 📨 mid-task updates are missed 🕵️ hidden state is lost ⚖️ conflicts are guessed through instead of clarified ✅ final-state verification is skipped 🔧 error detection and repair is failed

OSWorld 2.0 targets these bottlenecks directly: cross-source reasoning, implicit-state inference, multi-item tracking, conflict disambiguation, dynamic environments, streaming interaction, tutorial following, multimodal editing, visual-spatial precision, and proactive interaction.

Progress needs agents that can monitor change, preserve state, ask when evidence is insufficient, and catch mistakes before they propagate.

3d968253

LIKES33

XLANG NLP Lab@XLangNLP

🧵[5/8] The main result is not just a leaderboard. It is a cost–completion frontier.

On OSWorld 1.0, frontier agents are already near 83% accuracy. On OSWorld 2.0, the best 500-step setting still completes only 20.6% of tasks.

🏆 Claude Opus 4.8 reaches the best binary completion: 20.6% ⚡ GPT-5.5 is much more token-efficient: ~13.0% with only ~37K output tokens/task 🧠 Claude Opus 4.7 reaches 18.2%, but needs ~150K tokens/task 💸 Opus 4.8 pushes to 20.6%, but with ~224K tokens/task

The trade-off is steep: extra inference mostly buys partial progress, not reliable completion.

Strong agents can often get halfway through a workflow. The hard part is the last mile: preserving state, resolving conflicts, reacting to updates, and producing a correct final state. And as the task horizon grows, binary completion collapses. Long-horizon computer use is still very far from solved.

3d1.1K332

RETWEETS107

XLANG NLP Lab@XLangNLP

Two years ago, we built OSWorld 1.0 — the benchmark that became the standard for computer-use agents. Agents now score 83.5% on it. Problem solved?

Not even close.

🚀Today we introduce OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks.

What's new: 🎯 108 real-world workflows, each ~1.6 hours ⏱️ for a skilled human ⚙️ ~318 tool calls/task vs. ~30 in OSWorld 1.0 🌍 Grounded in authentic artifacts & stateful user profiles ⚡ Captures real phenomena: dynamic environments, streaming interaction, cross-source reasoning, implicit-state inference & more

📊 Best results: Claude Opus 4.8 reaches the highest accuracy at 20.6%, while GPT-5.5 is far more token-efficient but plateaus near 13%. No one is close to solving real computer use.

🏠 Homepage: https://osworld-v2.xlang.ai 📄 Paper: https://github.com/xlang-ai/OSWorld-V2/blob/main/OSWorld2.0.pdf 💻 Code: https://github.com/xlang-ai/OSWorld-V2 🤗 Dataset: https://huggingface.co/datasets/xlangai/osworld_v2_tasks

🧵 [1/8]

3d137.5K409173

REPLIES2

Labomen@labomen001

Is there a reason why you don't prevent cheating? Literally one of the first runs I opened, GPT almost immediately for some reason starts going the code way instead of just reading the PDF, but eventually it discovers the actual src of the local service (!). Step 62 is literally named "Retrieving secret key lines"

2d3011

XLANG NLP Lab@XLangNLP

🧵[3/8] Existing desktop benchmarks now risk looking close to solved. The tasks are often short, narrow, and self-contained, so high scores can hide whether agents can actually finish the end-to-end work users care about.

OSWorld 2.0 changes the horizon: ⏱ OSWorld 1.0 median human time: ~2 minutes ⏱ OSWorld 2.0 median human time: ~1.6 hours 📈 ≈48× longer by median time 📌 69.6% of tasks take skilled humans more than one hour 🛠 agent trajectories shift from ~30 steps in OSWorld 1.0 to >250 steps per task in OSWorld 2.0

This is workflow endurance, not isolated GUI control.

3d1.6K301

XLANG NLP Lab@XLangNLP

🧵[8/8] OSWorld 2.0 is built to make long-horizon computer-use evaluation realistic, inspectable, and reproducible.

We believe the next leap in computer-use agents won't come from better GUI clicking — it will come from agents that can read instructions, gather evidence, track state across hundreds of steps, resolve conflicts, and verify their own outputs before submitting.

OSWorld 2.0 is designed to measure exactly that gap — and to drive the field toward closing it.

We release everything to support the community:

🏠 Homepage: https://osworld-v2.xlang.ai 📄 Paper: https://github.com/xlang-ai/OSWorld-V2/blob/main/OSWorld2.0.pdf 💻 Code: https://github.com/xlang-ai/OSWorld-V2 🤗 Dataset: https://huggingface.co/datasets/xlangai/osworld_v2_tasks 📦 Trajectories: https://huggingface.co/datasets/xlangai/osworld2.0-trajectory 🧭 Trajectory Viewer: https://osworld-v2-monitor.xlang.ai

3d1K292

XLANG NLP Lab@XLangNLP

🧵[7/8] Contributions and Acknowledgments

Huge thanks to the amazing team behind OSWorld-V2 🚀 Led by @yuan_mengq43669, @adlsdztony1 & Xinzhuang Xiong, with@xhluca, @saa1605, @itsyuhao, @jiaqideng07, @xywang626, @DunjieLu1219, @BowenWangNLP, @vincentsunnchen and the whole XLANG crew making it happen.

Deepest gratitude to @taoyds for steering the project, and to our advisors @TianbaoX, @fredsala, @zhouyu, @ysu_nlp, @sivareddyg, @xwang_lk, @JustinLin610, @dayiheng_liu & @PengQi for their guidance throughout 🙏

We thank @SnorkelAI, our research & data partner, for their support of this work. We gratefully acknowledge support from the @GoogleResearch gift fund.

3d1.3K32

XLANG NLP Lab@XLangNLP

🧵[4/8] OSWorld 2.0 is designed to be broad enough to be real and controlled enough to score.

The 108 tasks span everyday and professional work across research & education, creative production, engineering & computing, personal services, administration & compliance, business & finance, and healthcare.

The benchmark covers: 🧾 authentic or adapted real-world artifacts: PDFs, spreadsheets, emails, receipts, forms, reports, videos, drawings, and portals 🧠 coherent stateful user profiles 🔄 controlled dynamic updates during execution 🙋 simulated user answers with bounded knowledge ✅ fine-grained final-state checks instead of only binary pass/fail 🔍 QA with unit tests, human re-solving, frontier-agent rollouts, reward-hacking audits, and false-negative audits

3d1.3K28

Nirmal Krishnan@nirmal_dist

@XLangNLP good chart and probably right

at this point, token efficiency matters just as much as pure accuracy

3d7529

Subramanya N@subramanya

@taoyds 318 steps is the right kind of scary. do you report recovery points when the run drifts?

3d7581

Nick Venturi@nickventuri

@XLangNLP gotta keep these bots humble before they start asking for a salary

2d1412

FateOfMuffins@FateOfMuffins

@XLangNLP But the graphs don't seem to show a plateau for GPT 5.5? It looks like it'll go up if you gave it more budget. The Claude curves are a lot more flat (which is what a plateau is)

2d2231

君子中庸@Chinese_XU

@XLangNLP I would like to ask a question: Are the agents participating in the test permitted to have prior learning regarding the task workflow?

2d1551

Qijun Han@sergiu2119655

@XLangNLP Wow, the trajectory viewing on the website is great!

3d3564

Noema@noemaclips

@XLangNLP @huybery Interesting one! @stalkermustang @xeophon @spicey_lemonade

3d331

ikka@Shahules786

@taoyds Interesting observation here, what techniques did you guys try?

3d1.1K1

Igor Kotenkov@stalkermustang

@noemaclips @XLangNLP @huybery @xeophon @spicey_lemonade already reading :)

3d641

Phi Browser@phibrowser

@XLangNLP that 20.6% tracks with the lived version. i'm fine for the first 30 tool calls. the wheels come off where i have to recall what i changed 200 calls back and the page state has drifted under me since. real computer use isn't a reasoning gap, it's a hold-state-for-hours gap.

3d154

Timur Yessenov@Timur_Yessenov

@XLangNLP This is the benchmark shape I trust more. ~318 tool calls vs ~30 means the hard part is not clicking; it’s staying sane after one bad step, one new email, and one contradictory receipt.

3d136

XLang NLP releases OSWorld 2.0 to test computer-use AI agents on complex, multi-hour workflows

Story Overview

Top models still fall short on marathon tasks

New test highlights where agents actually break