• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI
Reaction

AI peer-review prototype is claimed to find 90% of research 'nonsense' for a few hundred dollars in tokens

The builder says a cheap first pass targets the 1% of papers they believe carry 99% of the signal.

Casey HandmerCH
1 Source, 1h ago, first seen 1h ago

TLDR

A builder prototyping automated peer review claims current models can provide human-level reviews for about $0.10 in tokens, or rederive results, check statistics and audit cited papers for $5. They estimate auditing roughly 100 million papers would cost $500 million. Instead, they say they’re focusing on the 1% they believe carry 99% of the signal, and claim a cheap first pass can find 90% of the “nonsense” for a few hundred dollars in tokens.

Combined views

6K

1 Source, first seen 1h ago

236 likes25 comments39 saves16 reposts

Combined views

6K

1 Source, first seen 1h ago

236 likes25 comments39 saves16 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

1 Source

Casey Handmer@CJHandmerI've been prototyping an automated peer review project. With current models you get human level peer review for about $0.10 of tokens. For $5 you can rederive every result, check every statistical inference, and audit every paper cited in both directions. There are about 100m papers in the scientific corpus. Most are not important. Many are sloppy. For structural reasons it's essentially impossible to change published work, even to correct a typo. For $500m today you can thoroughly audit every paper ever written. Every p-hacked result, every plagiarist, every fabrication, every citation club, everything. I don't have $500m so instead I build a tree, focusing on the 1% of papers that carry 99% of the signal, doing a cheap first pass looking for obvious problems. I can find 90% of the nonsense for a few hundred dollars of tokens. Any guesses where the biggest surprises will be found?1h
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    1 Source

    Casey Handmer@CJHandmerI've been prototyping an automated peer review project. With current models you get human level peer review for about $0.10 of tokens. For $5 you can rederive every result, check every statistical inference, and audit every paper cited in both directions. There are about 100m papers in the scientific corpus. Most are not important. Many are sloppy. For structural reasons it's essentially impossible to change published work, even to correct a typo. For $500m today you can thoroughly audit every paper ever written. Every p-hacked result, every plagiarist, every fabrication, every citation club, everything. I don't have $500m so instead I build a tree, focusing on the 1% of papers that carry 99% of the signal, doing a cheap first pass looking for obvious problems. I can find 90% of the nonsense for a few hundred dollars of tokens. Any guesses where the biggest surprises will be found?1h
    Today's Rank

    #10

    Today's Rank

    #10