Ramp Labs@RampLabs
Today we’re releasing Ramp SWE-Bench: a private, production-grounded coding benchmark created from real engineering problems we've faced at Ramp.
10:26 AM · Jun 12, 2026 · 85.1K Views
Ramp launches Ramp SWE-Bench, a private coding benchmark built from real engineering challenges to evaluate AI on unseen tasks
Eric Glyman says frontier models train on public benchmarks.
7486927321105.3K
Sentiment
Many users praised Ramp's private SWE-Bench release for grounding AI coding benchmarks in real engineering problems, while some called it too easy, already saturated, or classic bubble behavior.
Pos
85.7%
Neg
14.3%
22 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS16.1KBOOKMARKS21LIKES111RETWEETS4REPLIES10
Eric Glyman@eglyman
public benchmarks are saturated. every frontier model has trained against them, and the leaderboard tells you near nothing.
we built ours from inside ramp — code no model has seen, graded against the bar our engineers ship to.
every company running on AI needs its own.
Ramp Labs@RampLabs
Today we’re releasing Ramp SWE-Bench: a private, production-grounded coding benchmark created from real engineering problems we've faced at Ramp.
2hViews 16.1KLikes 111Bookmarks 21
Ramp Labs@RampLabs
When measuring effectiveness versus cost, the frontier presents as a tradeoff rather than a single winner.
Read our methodology and explore the results below: http://labs.ramp.com/swebench
4hViews 3.3KLikes 34Bookmarks 11
Ramp Labs@RampLabs
Public benchmarks saturate quickly and inevitably leak into training data, with none quite resembling the work our engineers do every day. Building our own benchmark has allowed us to evaluate models within our own financial software ecosystem.
We compared models side by side and unearthed their behavioral differences. Head to head breakdowns available here: http://labs.ramp.com/swebench
4hViews 3.3KLikes 31Bookmarks 6
Ofir Press@OfirPress
@eglyman http://programbench.com http://codeclash.ai http://algotune.io http://swefficiency.com http://critpt.com http://videogamebench.com i think we have a few unsaturated public benchmarks let me know if you need more
Eric Glyman@eglyman
public benchmarks are saturated. every frontier model has trained against them, and the leaderboard tells you near nothing.
we built ours from inside ramp — code no model has seen, graded against the bar our engineers ship to.
every company running on AI needs its own.
1hViews 626Likes 27Bookmarks 7
Ofir Press@OfirPress
Ramp has a cool post about their new internal SWE-bench and they use mini-swe-agent as the harness powering the eval. DeepSWE also used mini-swe-agent as the harness, and showed that it performs just as well as Claude Code and Codex on the tasks.
Ramp Labs@RampLabs
Today we’re releasing Ramp SWE-Bench: a private, production-grounded coding benchmark created from real engineering problems we've faced at Ramp.
1hViews 1.8KLikes 6Bookmarks 5
Sergey Karayev@sergeykarayev
@RampLabs Great work! And if anyone would like to make their own personal SWE-Bench like this, based on real merged PRs into their codebase, we got you!
4hViews 583Likes 7Bookmarks 3
Max de Castro@maxricodecastro
@RampLabs a triple platinum banger!
4hViews 563Likes 8Bookmarks 1
Ofir Press@OfirPress
Read about mini-swe-agent in DeepSWE here: https://deepswe.datacurve.ai/blog#evaluation-harness
w/ @KLieret @jyangballin
Ofir Press@OfirPress
Ramp has a cool post about their new internal SWE-bench and they use mini-swe-agent as the harness powering the eval. DeepSWE also used mini-swe-agent as the harness, and showed that it performs just as well as Claude Code and Codex on the tasks.
1hViews 597Likes 0Bookmarks 1
Xiangyi Li@xdotli
@RampLabs we ported mini-swe-agent in opencode ui. might be relevant!
4hViews 518Bookmarks 1
joowon@n0w00j
@RampLabs ahahahahahaha Ramp the AI Lab fuck yea
2hViews 423Likes 4
Jack Joliet@jackjoliet
@RampLabs where's goat, Opus 4.6?
4hViews 694Likes 7
Ben Geist@b_geist
@RampLabs I’m personally a huge fan of this!
4hViews 476Likes 7
Synopticon@SynopticonAI
@RampLabs The rate at which benchmarks are saturated is accelerating across the board. Fable debuting at ~85% on day 1 is crazy!
3hViews 235Likes 3
Jeff@jeffdfeng
@OfirPress Any tooling recs for companies to build their own internal SWE-benches (tasks/dataset), beyond that harness?
1hViews 6Likes 1Bookmarks 1
joowon@n0w00j
@eglyman eric let me get more allocation pleaseeeeeeeeeeeeeee Ramp gonna go to 1 trillion at this point with AI lab
2hViews 436Likes 5
Xiangyi Li@xdotli
@RampLabs ramp is becoming another frontier lab eh
4hViews 263Likes 5
Daniel@daniel_li02
@RampLabs Ramp pivots to RL env company ‼️
4hViews 196Likes 5
Gabriel@gbrl_dick
@RampLabs @karimatiyeh this is great
2hViews 979Likes 3
Alex L@a_levitator
@RampLabs Benchmarks are the new moat
4hViews 263Likes 4
René Sultan@rene_sultan
@RampLabs That’s cool af
4hViews 201Likes 3
Load more posts