Managers excelled at guiding the agent through ambiguous tasks.
Management is an AI superpower. Superpowers by @obra is AI Management...
Another way of looking at this: if you know enough to specify the same software engineering task (all these folks did, regardless of occupation), then AI will just nail it (or not, or pretty much, or whatever).
If you don't... you get lost in the sauce.
Positive users highlight management and delegation skills as the key advantage when using Claude for coding tasks, while negative users call the study misleading due to task difficulty and selection bias issues.
No Digg Deeper questions have been answered for this story yet.
And here is the Anthropic report with the chart: https://cdn.sanity.io/files/4zrzovbb/website/b93c7465925dc052b9102209b29b58f11df4fe55.pdf
And this really is early evidence: sample size was excellent and approach to verifying outputs makes sense, but there are lots of unobservable elements that could play a role in relative success rates.
@emollick Managers are used to getting suboptimal work from employees and guiding them to improve it
That's the correct workflow for working with AI
It's the strong individual contributor who struggles because the AI's first draft is not as good as theirs would have been
@emollick Aha agreed!
I wrote a version of this argument here
@emollick Is the headline not how surprisingly even the results are? Management is just a few % points above law and that if you don't include judged success
@emollick Now we can finally test if management books transfer positive skills.
I've seen this firsthand. I know engineers who are 1000x better than me at Python, CSS, scripts, frontend — but most haven't had to manage people. And managing people is exactly this: getting the right thing across to different personalities so the context survives, the goal stays visible, and everyone knows what to do.
Working with agents, it's the same skill. The model isn't the bottleneck — being able to say what you want and what "good" looks like, to something that doesn't live in your head, is.
@johnennis @emollick Pretty much. My biggest frustration is the verbose feedback loop instead of just fixing things. AI is powerful and cheap, but frustrating because unlike a human doesn't actually learn.
@emollick maybe, but specs don’t save messy taste
@emollick This study is misleading, it doesn't classify the difficulty of the task.
We assume everyone was doing things that were equally as challenging for models & operators.
The study also acknowledged that managers tend to confirm/reward success therefore making it easier to classify
Spot on. Management is the real superpower here — not just prompting, but defining what ‘good’ and ‘safe’ actually look like. In my setups the agent can plan and read autonomously, but any write/deploy/payment action is scoped to a separate process with a human gate. An LLM will get socially engineered eventually. Curious how you’re thinking about the accountability layer as agents get more autonomous?
@emollick you named three levers - better instructions, catching the error, correcting it - and in that same piece credits all of it to subject-matter expertise. then post it as management.
100% agree. I’ve found that lots of management techniques can be used to manage agents - from defining “what success looks like” to asking the right questions to stress test agent’s work and run retroactive (“how would you change the way you work so this mistake doesn’t happen again”).
@emollick We’ve been touting the value of disintermediation of app development since Lotus 1-2-3 first added a macro language to their spreadsheet.
@emollick Knowing exactly what good output looks like was always the actual skill.
The code was never the hard part for the people who managed it well.
@emollick management was always the skill
@emollick my takeaways General intermediate understanding of everything is good enough to get good output from AI . Specialists are still needed at the tail end for verification.
domain expertise > technical skills
@emollick The tell is how someone delegates to people. Managers who hand a human a vague ask and hope for the best do the same to an agent, then blame the model. The ones who win write a real spec, a definition of done, and two examples of what bad output looks like. Same muscle, new tool.
@emollick It’s that, or the fact that managers have years more experience and hence better judgment on evaluating what llms produce.
@emollick It makes sense because they understand the full workflow. Might be worth checking middle management numbers to see if AI is already shifting people from lower roles into management