I’m extremely excited about Replication Radar, built by Rhea Karty at Harvard’s lab and supported by @cosmos_inst & @TheFIREorg.
It’s close to an idea I’ve been a little obsessed with lately: the “knowledge crawler.”
The basic idea: use AI to crawl as much of human knowledge as possible — papers, books, claims, citations, replications, retractions, old debates, buried null results — and ask the annoying but essential questions.
Does this actually hold up?
Did this famous study replicate?
Is this field resting on three papers everyone cites but nobody has checked in 20 years?
Was this “settled” conclusion ever actually settled?
Are there forgotten papers that were right too early, too unfashionable, or just too boring to get attention?
This would be a gigantic undertaking. Access to scholarship, copyright, licensing, academic incentives, institutional defensiveness — all of it would be hard.
But hard is not the same as impossible. And this is worth doing.
And yes, people will say, “Sure, maybe for science. But what about the humanities?”
Well, a lot more of the humanities than people admit can be reduced to factual claims.
What happened? Who said what? Did this policy produce that outcome? Did this institution actually do what people claim? Did this theory predict anything, or just explain everything after the fact?
Those claims can be tested too. Not perfectly. Not by a magic TruthBot. But tested.
That’s the whole point.
We don’t really know most things are “true” in some final sense. We know what has survived serious attempts to prove it false.
Human knowledge is overwhelmingly a project of subtraction.
You get closer to truth by removing error. Bad data. Fraud. Wishful thinking. Failed replications. Citation circles. Beautiful theories reality refuses to cooperate with.
Yes, my ambition here is huge. Fine. It should be.
A project like this might once have taken a century. With AI, maybe we can get a much clearer map of what we know, what we only think we know, and what most urgently needs to be researched next in a handful of years.
Will it show that we know a hell of a lot less than we think?
Almost certainly.
Good. That’s progress.
Proud that @TheFIREorg and @cosmos_inst are helping push this kind of truth-seeking work forward.