/Tech1d ago

SWE-bench co-creator John Yang launches two simulator-backed CodeClash arenas to evaluate AI agents on logistics and cybersecurity

Agents negotiate supply-chain contracts and defend simulated drone networks

435245.4K

#78

Original post

John Yang@jyangballin#1244inTech

I'm really excited about the amazing work @Muhtasham9 has been doing on CodeClash.

Our initial release consisted entirely of LMs competing in games.

Since then, we've found a ton of coding environments that can be repurposed as competitive arenas. Leaderboard soon.

muhtasham@Muhtasham9

CodeClash is moving beyond games ⚔️

We added 2 simulator-backed arenas where LMs compete by improving codebases:

SCML: negotiate supply-chain contracts to maximize profit CybORG: defend simulated drone networks in a CAGE Challenge environment

Same CodeClash loop. New objectives.

11:09 AM · Jun 23, 2026 · 2.3K Views

Sentiment

Users praise CodeClash as one of the most underrated benchmarks thanks to its new simulator arenas for LLM code competitions and improvement tasks.

Pos

100.0%

Neg

0.0%

1 comments with sentiment.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Posts from X

Most Activity

muhtasham@Muhtasham9

Current coding evals ask:

“Can you fix this bug?” “Can you write this function?”

But real engineering is goal-directed:

increase profit defend simulated systems allocate resources adapt to competitors improve over time

So CodeClash arenas test whether code wins on outcomes.

1d124

LIKES2RETWEETS1

Qiuyang Mang@MangQiuyang

@jyangballin @Diyi_Yang @Muhtasham9 After talking with @OfirPress several times, I think CodeClash is one of the most underrated benchmarks out there.

It offers a very natural setting for studying long-horizon Elo-rating scaling laws, similar to what we discussed in our recent blog: https://joyemang33.github.io/blog/2026/humans-dont-just-sample/

1d192

REPLIES1

muhtasham@Muhtasham9

Games are useful because they’re clean and measurable.

But simulators let CodeClash test messier objectives:

profit resilience resource allocation adaptation

That’s closer to the kinds of outcomes real software is written to achieve.

1d431

muhtasham@Muhtasham9

SCML tests supply-chain strategy.

Agents write SCML OneShot negotiators that buy and sell through contracts across simulated worlds.

The code has to do more than run.

It has to make good decisions against other agents.

1d901

muhtasham@Muhtasham9

CybORG tests defensive decision-making.

Agents write blue-team policies in a simulated cyber environment and compete on reward.

The goal is controlled cyber-defense benchmarking, not open-ended offensive capability.

1d241

muhtasham@Muhtasham9

More arenas are in progress too:

ABIDES: financial-market simulation Bomberland: competitive grid-world strategy

If you know a simulator where: - agents control code policies - multiple agents interact - outcomes are measurable - runs are reproducible

it might make a great CodeClash arena.

Repo: https://github.com/CodeClash-ai/CodeClash

1d402

muhtasham@Muhtasham9

CodeClash is moving beyond games ⚔️

We added 2 simulator-backed arenas where LMs compete by improving codebases:

SCML: negotiate supply-chain contracts to maximize profit CybORG: defend simulated drone networks in a CAGE Challenge environment

Same CodeClash loop. New objectives.

1d3.2K151

Ferbin@Ferbin08

@jyangballin @Muhtasham9 Will the same models dominate every environment or does each one reward different approaches?

1d51