• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI

Discoveries versus activity as measures of AI's impact

One post argues for judging AI more by total exploits discovered, math problems solved and algorithmic efficiency, and less by token spending, code volume or self-reported speedups.

Ethan MollickEM
rishiRI
Nat McAleeseNM
6 Sources, 19d ago, first seen 19d ago

TLDR

A September 20 post argues that inputs and intermediate measures can be hard to interpret: activity can change without outputs changing, or vice versa. The author favors more focus on aggregate outputs, while noting that experiments are harder to run and changes can be difficult to attribute to AI. In an August 18 post, the same author shared tentative conclusions about discovery trends: sharp acceleration in cyber, some acceleration in math and no clear acceleration in algorithms.

Combined views

63.6K

6 Sources, first seen 19d ago

451 likes34 comments150 saves52 reposts

Combined views

63.6K

6 Sources, first seen 19d ago

451 likes34 comments150 saves52 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

6 Sources

tom cunningham@testinghamWhat impact is AI having? I'm of the view that we should be focussing more on aggregate outputs, relatively less on inputs or intermediate proxies. - Inputs: time spent across different activities; money spent on tokens. - Intermediate proxies: working papers; lines of code, commits & pull requests; self-reported speedup. - Final outputs: total cyber exploits discovered; total math problems solved; algorithmic efficiency. The disadvantage of inputs & intermediate proxies is that they're very hard to interpret -- totally possible that they move, but outputs don't, and vice versa. Lines of code could explode without value changing; and vice versa. A disadvantage of final outputs is (1) it's harder to run experiments; (2) it's hard to attribute to whether the change is AI or not. But in some domains the trend-break is so large that it's *obviously* AI.19d
Ajeya Cotra@ajeya_cotraRT @testingham: What impact is AI having? I'm of the view that we should be focussing more on aggregate outputs, relatively less on inputs…19d
rishi@RishiBommasaniWe need more initiatives that specialize in the measurement of outcomes that are deeply immerse in frontier AI. Most frontier AI orgs study (or build) the production function as Tom mentions. And more downstream measurement orgs are just plodding along as they have for many years, often none the wiser that changes may be coming to their work because of upstream frontier AI adoption. There are efforts under way to fund in this space (e.g. run lots of RCTs on firm-level productivity effects from frontier AI adoption). But I still believe the growth rate in this broad meta-category is not fast enough for the scale of cross-cutting transformation I would frontier AI will deliver. On methods, these initiatives should find ways to solve attribution problems as simply as possible. We can use fancy causal methods when that's the best we can do with the data we have, but if we can just directly say stuff is caused by AI because we know it is, or see such large effects that no other explanation is viable, that's even better. (Notably, intellectual fixation sometimes among economists and social scientists on fancier ways of establishing causality runs counter to the agenda of realtime sensemaking.)19d
Ethan Mollick@emollickI would broaden this to all social science. We are in uncharted waters. We need fast, smart research on AI that is deeply informed by AI's abilities, is forward-looking, and may not be fully nailed-down. This sort of work is not usually high status in fields, but it is critical.19d
Nat McAleese@__nmca__pre-registering a bunch of trends that we will later check for ai impact would be stellar18d
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    6 Sources

    tom cunningham@testinghamWhat impact is AI having? I'm of the view that we should be focussing more on aggregate outputs, relatively less on inputs or intermediate proxies. - Inputs: time spent across different activities; money spent on tokens. - Intermediate proxies: working papers; lines of code, commits & pull requests; self-reported speedup. - Final outputs: total cyber exploits discovered; total math problems solved; algorithmic efficiency. The disadvantage of inputs & intermediate proxies is that they're very hard to interpret -- totally possible that they move, but outputs don't, and vice versa. Lines of code could explode without value changing; and vice versa. A disadvantage of final outputs is (1) it's harder to run experiments; (2) it's hard to attribute to whether the change is AI or not. But in some domains the trend-break is so large that it's *obviously* AI.19d
    Ajeya Cotra@ajeya_cotraRT @testingham: What impact is AI having? I'm of the view that we should be focussing more on aggregate outputs, relatively less on inputs…19d
    rishi@RishiBommasaniWe need more initiatives that specialize in the measurement of outcomes that are deeply immerse in frontier AI. Most frontier AI orgs study (or build) the production function as Tom mentions. And more downstream measurement orgs are just plodding along as they have for many years, often none the wiser that changes may be coming to their work because of upstream frontier AI adoption. There are efforts under way to fund in this space (e.g. run lots of RCTs on firm-level productivity effects from frontier AI adoption). But I still believe the growth rate in this broad meta-category is not fast enough for the scale of cross-cutting transformation I would frontier AI will deliver. On methods, these initiatives should find ways to solve attribution problems as simply as possible. We can use fancy causal methods when that's the best we can do with the data we have, but if we can just directly say stuff is caused by AI because we know it is, or see such large effects that no other explanation is viable, that's even better. (Notably, intellectual fixation sometimes among economists and social scientists on fancier ways of establishing causality runs counter to the agenda of realtime sensemaking.)19d
    Ethan Mollick@emollickI would broaden this to all social science. We are in uncharted waters. We need fast, smart research on AI that is deeply informed by AI's abilities, is forward-looking, and may not be fully nailed-down. This sort of work is not usually high status in fields, but it is critical.19d
    Nat McAleese@__nmca__pre-registering a bunch of trends that we will later check for ai impact would be stellar18d
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet