Engineer Uses AI Agents To Quiz Code Changes And Build Micro-Worlds

life hack: before starting a work session, tweet about whatever you'll be working on and stir up some conversation

then, whenever you reflexively check twitter, you'll end up still working on the thing instead of wasting time 😂

btw talk will is at AI Engineer World's Fair on Wed, 10:45am. come by if you'll be there!

https://www.ai.engineer/worldsfair/schedule?day=3&session=asn_slot_2026_07_01_breakout_track_06_1545_2026_06_03t11_33_40_915z

Also I should clarify: I don't mean "quiz me on the requirements" although that is a great practice.

I mean "quiz me on whether I have correctly understood how it works, after it's done"!

Which is weirder but insanely valuable.

@ario yeah great topic! i think often an objection to understanding is something like "many programmers don't need to understand memory layout anymore and it's fine"...

and the reality is, everyone has finite capacity for understanding so we have to make tradeoffs

mm yeah. one way i like to think about is that you're constraining a specification space which is usually a bit underspec-d / vague in some way. and then the agent should help explain where it landed in that space and why: what are the dimensions, what choices did it make.

it can do this either before the task (asking you questions) or afterwards (telling you what important choices it filled in)

personally i find that asking questions before works very well w/ modern models, but they're not always very good at surfacing what important choices they made after the fact. curious if you agree.

@geoffreylitt I do both of those too, but my goal is very much so to do these when planning code changes, and after the changes are made, so as to stress test where the agent made assumptions that are wrong, or went ahead with the work with low confidence. Curious how you solve these problems

@stevekrouse I’m just spreading the gospel of Steve here

@geoffreylitt Would love to be a fly on the wall and attend if there’s a spot. Noticed this with the initial Claude Artifacts

@geoffreylitt Calibrating the ROI on understanding. Some kind of rubric that provides as a compass for when a deeper understanding pays off vs a waste of time

@oznova_ This is my full code explainer prompt

https://gist.github.com/geoffreylitt/a29df1b5f9865506e8952488eac3d524

@geoffreylitt Great title

@geoffreylitt curious what you do when the agent passes the quiz but still confidently gets it wrong. that's the harder case than obvious errors.

@oznova_ This is my quiz prompt -- works surprisingly well in my experience as long as you use Claude models!

Only took a few iterations to get this to "good enough", I'm sure it could be made much better.

My evals is my coworkers saying "this skill rocks" :P

i see this as two understandings - understanding what it does (including all the corner cases) and understanding how it does it. i see the former as being really valuable in the feedback loop during design - i worry less about the latter because that is easier to fix if the "what it does" is defined correctly and there aren't any latent conflicts between human and model.

@geoffreylitt How do you ensure good quiz questions? Do you change the prompt much between domains? Have you iterated on it much? Any evals?

@geoffreylitt will the talk be recorded? or will you be sharing the slides?

@alexanderbenz the quiz is for me not the agent :)

@geoffreylitt This is a good idea!

@geoffreylitt your app barely even works properly, none of this is beneficial