Garry Tan Shares Workflow For Self-Improving AI Agents With Progressive Evals

By evals I mean literally tell the agent: given what we discussed about what we are doing and why and what happened, use three different frontier models to look at inputs and outputs of your skill file calling the code, and rate it on effectiveness. Why isn’t it a 10? How could it be made to be so?

Run this a few times and you will be surprised how fast it gets astonishingly better

And since it is in a skill file plus code with evals (LLM as judge) and unit tests, it stays better forever

I don't use OpenClaw but use a 'Lite' version via Codex which can control your computer with very little config.

I'm flying back from Vietnam and got it to look at my plane ticket, and work out some options to get back home via train/bus with some tabs open ready to book.

Very impressive!

@garrytan Hermes is the easiest to use agent

@garrytan Everyone is focused on the model, but the real leverage is in the evaluation loop. Nice setup.

@garrytan @garrytan Can you make Jelly the crab the mascot of Ycomb PLEASEEEEEEE

@garrytan They just keep getting better and better

@garrytan The underrated part is the eval loop

Most people use agents like interns The real jump is treating them like batch processors with taste checks at every size increase

@garrytan the loop closing on itself is the real unlock.

not AI that does tasks.

AI that gets better at tasks without being told to