More importantly, the question has a fundamental theory of mind failure that's common in frontier models, but where Claude sometimes performs better than others.

It writes its preferred analysis into the question.

4/

@emollick I aksed for a question in my field - combining learning on networks with network formation - and got this mediocre output which I'd give a C or C-

Way too easy for a PhD class on this topic, and generally not good for testing understanding of anything.

Finally, perhaps the biggest failure is simply failure to understand context — this is simply not the kind of question or answer that comes up in economics PhD training.

We don’t read the old texts and write essays synthesizing them. We write and solve models.

6/

Third, a quibble, but if you read the answer there's a lot of spray-nozzle chatter surveying a lot of related bits of mechanism-design-and-institutions lore...

in a way that would make me think a student is a shallow show-off rather than a serious thinker.

5/

Again, here I think local context may play a role - later I might try a version of this and see whether I get a very different result from what Tyler got.

7/7

cc @emollick

@ben_golub @emollick Fun exercise. I'm curious - Why "pose a question" rather than "answer a question"?

I'd think most people are more interested in having LLMs answer questions, but the userbase of higher-cost frontier models is certainly not "most people".

@ben_golub the prompt did ask for "not a math question", whatever that means

@idio_vol @emollick My sense is it's where the frontier is.

Asking good questions is a sign of very high expertise.

@ben_golub I thought the biggest failure was Claude’s failure to inform Tyler that his comment about information and transaction costs is utterly wrongheaded.