Mitchell Hashimoto, Ghostty creator, finds an AI agent optimized a Go renderer to 2ms, while his naive implementation achieved 0.02ms

VIEWS54.2KREPLIES25

Sensible voices are (finally) breaking through with AI. @mitchellh one of the best ones.

“I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.”

Read it 👇

I've got an agent in a loop optimizing a renderer with the goal to minimize frame times (and tests to measure). It got times down from 88ms to 2ms and allocations down from ~150K to 500. Sounds good, right? Wrong. This is exactly why agent psychosis is a big fucking problem.

As an experiment, I rewrote the Ghostty core render state in Go, with access to identically laid out data structures as Ghostty and the exact same validation tests. I made a purposely naive renderer (simple, correct, but slow). 88ms per frame with 150,000 allocations (horrendous, lol)!

I then kickstarted a Ralph loop to bring the frame times down. I told it it can't modify input data structures or the public API or tests (they're correct), but it can do anything else it wants. It got to work.

It has worked for about 4 hours. I've spent around $350 on this experiment so far. The results?

88ms => 1.5ms 150K allocs => ~500 allocs

Incredible right? Nope.

My hand-written renderer I ported has frame times (same benchmark) of ~20us (0.020ms) and 0 allocations in the update path.

This is the problem with psychosis and lacking systems understanding. If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better on throughput.

The people who blindly trust agent output are in the former camp. They're sheeple, overdrinking from a fountain of mediocrity.

Standard disclaimer: I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.

34d54.2K567101

BOOKMARKS157LIKES625

kache@yacineMTB

"If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better"

precisely what I have been saying

Mitchell Hashimoto@mitchellh

I've got an agent in a loop optimizing a renderer with the goal to minimize frame times (and tests to measure). It got times down from 88ms to 2ms and allocations down from ~150K to 500. Sounds good, right? Wrong. This is exactly why agent psychosis is a big fucking problem.

As an experiment, I rewrote the Ghostty core render state in Go, with access to identically laid out data structures as Ghostty and the exact same validation tests. I made a purposely naive renderer (simple, correct, but slow). 88ms per frame with 150,000 allocations (horrendous, lol)!

I then kickstarted a Ralph loop to bring the frame times down. I told it it can't modify input data structures or the public API or tests (they're correct), but it can do anything else it wants. It got to work.

It has worked for about 4 hours. I've spent around $350 on this experiment so far. The results?

88ms => 1.5ms 150K allocs => ~500 allocs

Incredible right? Nope.

My hand-written renderer I ported has frame times (same benchmark) of ~20us (0.020ms) and 0 allocations in the update path.

This is the problem with psychosis and lacking systems understanding. If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better on throughput.

The people who blindly trust agent output are in the former camp. They're sheeple, overdrinking from a fountain of mediocrity.

Standard disclaimer: I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.

34d49.6K625157

RETWEETS323

Mitchell Hashimoto@mitchellh

I've got an agent in a loop optimizing a renderer with the goal to minimize frame times (and tests to measure). It got times down from 88ms to 2ms and allocations down from ~150K to 500. Sounds good, right? Wrong. This is exactly why agent psychosis is a big fucking problem.

As an experiment, I rewrote the Ghostty core render state in Go, with access to identically laid out data structures as Ghostty and the exact same validation tests. I made a purposely naive renderer (simple, correct, but slow). 88ms per frame with 150,000 allocations (horrendous, lol)!

I then kickstarted a Ralph loop to bring the frame times down. I told it it can't modify input data structures or the public API or tests (they're correct), but it can do anything else it wants. It got to work.

It has worked for about 4 hours. I've spent around $350 on this experiment so far. The results?

88ms => 1.5ms 150K allocs => ~500 allocs

Incredible right? Nope.

My hand-written renderer I ported has frame times (same benchmark) of ~20us (0.020ms) and 0 allocations in the update path.

This is the problem with psychosis and lacking systems understanding. If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better on throughput.

The people who blindly trust agent output are in the former camp. They're sheeple, overdrinking from a fountain of mediocrity.

Standard disclaimer: I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.

34d601.9K7.7K1.9K

Shreya Shankar@sh_reya

This. It feels a bit unsettling in systems academia. Worse yet, the problems many papers try to optimize are already well-represented in the scientific literature, and even a good (albeit agent psychosis-induced) result often stems from the LLM having been trained on the solution.

I am absolutely in favor of pushing AI to solve hard problems, but I worry we are missing the plot. Maybe the current trend of writing papers using AI to (re)discover systems algorithms is on the path to AI truly discovering novel systems optimizations. If not, I hope we correct course soon.

Mitchell Hashimoto@mitchellh

I've got an agent in a loop optimizing a renderer with the goal to minimize frame times (and tests to measure). It got times down from 88ms to 2ms and allocations down from ~150K to 500. Sounds good, right? Wrong. This is exactly why agent psychosis is a big fucking problem.

As an experiment, I rewrote the Ghostty core render state in Go, with access to identically laid out data structures as Ghostty and the exact same validation tests. I made a purposely naive renderer (simple, correct, but slow). 88ms per frame with 150,000 allocations (horrendous, lol)!

I then kickstarted a Ralph loop to bring the frame times down. I told it it can't modify input data structures or the public API or tests (they're correct), but it can do anything else it wants. It got to work.

It has worked for about 4 hours. I've spent around $350 on this experiment so far. The results?

88ms => 1.5ms 150K allocs => ~500 allocs

Incredible right? Nope.

My hand-written renderer I ported has frame times (same benchmark) of ~20us (0.020ms) and 0 allocations in the update path.

This is the problem with psychosis and lacking systems understanding. If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better on throughput.

The people who blindly trust agent output are in the former camp. They're sheeple, overdrinking from a fountain of mediocrity.

Standard disclaimer: I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.

33d9.9K6418

Mitchell Hashimoto@mitchellh

@kalin_t Yes, there is value in that for sure. But what happens to the S-tier programmers giving up? Who becomes the next S-tier programmers if they never put in the work?

34d7.5K1534

kache@yacineMTB

@mitchellh yes!

Mitchell Hashimoto@mitchellh

I've got an agent in a loop optimizing a renderer with the goal to minimize frame times (and tests to measure). It got times down from 88ms to 2ms and allocations down from ~150K to 500. Sounds good, right? Wrong. This is exactly why agent psychosis is a big fucking problem.

As an experiment, I rewrote the Ghostty core render state in Go, with access to identically laid out data structures as Ghostty and the exact same validation tests. I made a purposely naive renderer (simple, correct, but slow). 88ms per frame with 150,000 allocations (horrendous, lol)!

I then kickstarted a Ralph loop to bring the frame times down. I told it it can't modify input data structures or the public API or tests (they're correct), but it can do anything else it wants. It got to work.

It has worked for about 4 hours. I've spent around $350 on this experiment so far. The results?

88ms => 1.5ms 150K allocs => ~500 allocs

Incredible right? Nope.

My hand-written renderer I ported has frame times (same benchmark) of ~20us (0.020ms) and 0 allocations in the update path.

This is the problem with psychosis and lacking systems understanding. If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better on throughput.

The people who blindly trust agent output are in the former camp. They're sheeple, overdrinking from a fountain of mediocrity.

Standard disclaimer: I use AI all the time. I like AI. The point I'm making is to not blindly accept results. Think. Analyze. Learn.

34d5.1K1021

kache@yacineMTB

You need to have standards for software that come from what your computer is actually capable of doing. How much time does it take, at the limit, to render something? How much time does it take to update the state, move data to gpu, and to draw a texture? That is your budget

kache@yacineMTB

"If you don't understand the system, you're going to accept that this is an incredible result. If you understand the system, you'll see better solutions immediately and can do roughly 75x better"

precisely what I have been saying

34d3.7K585

kalin@kalin_t

While I agree with your general sentiment there's something novel about the fact that someone who does _not_ understand to this degree can still get several orders magnitude improvement for 'free'. Or at least for 350$.

The world clearly failed at convincing most software to be well written so I'll take some freebie gains of the garbage software I have to suffer with in day to day life.

34d9.3K693

Mitchell Hashimoto@mitchellh

@markerdmann Yes basically, you can see the Zig version here: https://github.com/ghostty-org/ghostty/blob/main/src/terminal/render.zig I'll post the Go one eventually, I want to do more tests first. It allocates on first run, but with the workloads we're testing, doesn't need to for subsequent (frees/allocs into a buffer).

34d1.5K159

The Hypochondriac Warlord@mojito_flow

@mitchellh That's probably the worst use case for an LLM and I doubt it'll ever be good at that.

LLM's are for rapidly churning out internal apps with mediocre performance that nobody cares about. If you're doing anything even remotely complicated or with performance reqs its useless.

34d9.4K432

Mitchell Hashimoto@mitchellh

@doctorscm Its not a dichotomy. What you said is definitely true, but S-tier developers are basically giving up their brains over this. And the problem is if you do it enough, you start losing enough systems understanding to even manually get in deep for the parts that matter.

34d2.3K242

Mitchell Hashimoto@mitchellh

@kunchenguid If we assume a linear cost, that's going to be $3500. Who is spending $3500 per feature?

34d4.4K31

Mitchell Hashimoto@mitchellh

@mojito_flow Yep, they're excellent for that.

34d7.4K331

gfodor.id@gfodor

@mitchellh I feel like if my performance target was 10ms here then obviously vibing to 2ms with zero marginal cognitive load on my end is the right choice in many scenarios. I think the future models are going to inflate away the tech debt.

34d1.4K27

Mitchell Hashimoto@mitchellh

@markerdmann Read the commit history and find the PRs where it was introduced plus bugs, its an interesting piece of engineering!

34d27993

kalin@kalin_t

@mitchellh Yeah definitely worth encouraging. The optimist in me hopes AI can be the infinity-knowledge teacher to help make more S-tier programmers, if they have the drive.

34d1.6K72

Perry E. Metzger@perrymetzger

So, it’s one thing to say “but this wasn’t good enough for me, because the AI didn’t have real insight into the problem, and I managed far better”, that’s a totally valid point, but another to say that accepting an improvement of that size is “psychosis”. Not every programmer is as good as you are, and many of them would never have had the insight and know how to completely restructure the problem in the first place, and for them, getting a factor of 40x improvement would be pretty great.

Most software isn’t Ghostty or Vault either. Most of it is boring crap, and working at all and mostly correctly is enough. For every Ghostty, there are a hundred office billing systems maintained by people who could never, on their very best day, have improved the performance of Ghostty’s rendering at all.

This is no longer even “OK, so the dog only plays chess as well as a rank beginner”, this is now “the dog only plays chess as well as an above average chess player”.

I’ve recently been refactoring a one million CLOC C codebase. I am pretty decent and could have done the job alone over a long period of time, maybe a couple of years or more, but combining my abilities with some agents, I have made dramatic progress in six weeks. And yah, I constantly have to tell the robots they’ve solved the wrong problem, but so what, I’ve had to have that conversation with human programmers too. Meanwhile, I bet by this time next year that will be a hell of a lot better too.

34d446131

gfodor.id@gfodor

@mitchellh Also fyi opus 4.7 is basically braindead compared to gpt 5.5 for this kind of work

34d97581

Mitchell Hashimoto@mitchellh

@NeonNoodle22 I'm willing to spend $500 on this and its still running so we'll see. I forgot to include but another point is that most people (incl me) are not spending $500 per feature lol.

34d1.2K11

Conrad Kramer@conradev

@mitchellh If you simply told the agent that 500 was not good enough, could it not get it down to 0? did the agent's work inform your opinion on how to get it to 0?

I don't see how a performance improvement is LLM psychosis unless the model or human claimed it could do no better?

34d2.5K21