Shreya Shankar argues a Claude Code study circularly defines novice users to claim non-programmers succeed with the tool

I feel like this table is misleading. (I) It is not that novices write underspecified tasks; their methodology classifies underspecified prompts as “novice.”(ii) Every time a frontier lab says “expertise in the task,” they implicitly mean “expertise in the task AND the relevant data to analyze.” This assumption is trivially ok when there is no data to analyze, like in many coding tasks, but it’s unreasonable to assume people to be an expert in the specific dataset at hand. There are always ambiguities in real world data. Human data scientists take months to onboard. AI data scientists take $100k to build understanding of an enterprise-scale dataset (ie “semantic layer”), and still are no good.

The North Star for AI data analysts is to assist people who want to make data-driven decisions but don’t know what’s in their (real, messy) data. So perhaps it is a contrarian take for me to say that this “novice” prompt should actually work…and we have a long and exciting way to go to make this happen