Geoffrey Litt argues AI coding agents struggle with UI because specification and implementation tasks constantly blur

I struggled and only broke through once my skill system (which I like to think as part of my agents habitat) reached a point where how I think/work is pretty closely followed by claude

I switched to vertical tabs and give focus only to one while the others work. Then cycle.

I think the reason I don’t feel the frustration anymore is because my skills fully reflect my thinking

@jamescham I have tried many of these structures. In my domain, the main problem is mostly that they can't see and understand highly interactive interfaces well enough to reliably validate their work. When the models can't fully close their own loops, a lot of these strategies don't work.

@andy_matuschak both happy camps keep attention single-homed: in (a) you are the one thread, in (b) one other thing fully owns you while agents run offstage. the middle asks you to half-watch several loops at once, and partial attention split across threads isnt weaker focus, its no focus

@andy_matuschak After spending a few months trying to keep as many parallel agents going as possible concurrently, I’ve now settled on a chief-of-staff style delegation pattern, where I spend 95% of my time working with one agent, who I use to scope jobs-as-context-packets for subagents

@puresight Book? It's a Twitter thread!

@andy_matuschak 😢 Exactly.

@andy_matuschak @jamescham i also am very skeptical of claims for long running agents; the best loop is two agents debating each other and i am the final judge with relatively short bursts. anything more long ranging breaks down unless its a trivial problem

@andy_matuschak I’m here too. I experimented quite a bit with Cursors agent interface but found it unsatisfying because of this I think. I have found the closest middle ground to be highlighting a large chunk of code (e.g. a function and delegate to agent) while I think through next steps

@andy_matuschak @jamescham If you’re focused more on UI then yea I think that’s tough and maybe the best path is to just take a few automated looping shots at building a prototype of what you want, work on it to get it right and then queue up another agent to go back and rebuild it but with good code.

"teams of agents making technical plans, reviewing each others' work, autonomously and adversarially testing, etc. You still get robust "technical oversight"—just by other agents."

i think this doesn't work either. i'm somewhere in the middle ground too, and the primary reason is that i think it makes me feel connected to the codebase and understand the nuances and design decisions that went into it. agents reviewing the plans, etc doesn't really solve this for me

@andy_matuschak so glad to hear you talk about this, it absolutely sucks. and I don’t have an answer. but the lieutenant pattern seems like a promising approach

@andy_matuschak you would love what @humanlayer_dev do.

For me the nature of the work dictates the loop for the reasons you mentioned in the end. It’s wildly frustrating to try and get an agent to make a smooth, beautiful, interactive transition/UI exactly how I want, and I haven’t managed to delegate the product and its design/behavior yet.

Totally different for infra/backend

@blelbach @andy_matuschak The problem is largely unattended loops is that the output tends to be larger than people can read while staying 100% focused.

If the agent could tackle very small but complex parts of the codebase and work autonomously on these for a long period of time, id do that.

What delegated slow loops have you found highly satisfactory experiences from? For me it’s either just

1. Translating a very clear spec, or from one language to another. Essentially some task with low room for variance. 2. Optimization tasks like auto research

Just about nothing else yet

@puresight Oh, I see your point. Yes, I fear we are moving too fast for books. And for expertise! The "skills" I build up here become irrelevant in months as models change.

@andy_matuschak This echoes my experience with self driving, as it went from partial to fully autonomous.

Driving is fun, not driving at all is fun, sorta driving is strressful.

@abhinavsharma For me: relatively little. Translating an app from SwiftUI to React Native, lol.

But for friends, it’s often low stakes prototyping. “Hey, wouldn’t it be cool if…” kind of stuff, iterated in the background over days.