Séb Krier#667
I’m extremely excited about Replication Radar, built by Rhea Karty at Harvard’s lab and supported by @cosmos_inst & @TheFIREorg.
It’s close to an idea I’ve been a little obsessed with lately: the “knowledge crawler.”
The basic idea: use AI to crawl as much of human knowledge as possible — papers, books, claims, citations, replications, retractions, old debates, buried null results — and ask the annoying but essential questions.
Does this actually hold up?
Did this famous study replicate?
Is this field resting on three papers everyone cites but nobody has checked in 20 years?
Was this “settled” conclusion ever actually settled?
Are there forgotten papers that were right too early, too unfashionable, or just too boring to get attention?
This would be a gigantic undertaking. Access to scholarship, copyright, licensing, academic incentives, institutional defensiveness — all of it would be hard.
But hard is not the same as impossible. And this is worth doing.
And yes, people will say, “Sure, maybe for science. But what about the humanities?”
Well, a lot more of the humanities than people admit can be reduced to factual claims.
What happened? Who said what? Did this policy produce that outcome? Did this institution actually do what people claim? Did this theory predict anything, or just explain everything after the fact?
Those claims can be tested too. Not perfectly. Not by a magic TruthBot. But tested.
That’s the whole point.
We don’t really know most things are “true” in some final sense. We know what has survived serious attempts to prove it false.
Human knowledge is overwhelmingly a project of subtraction.
You get closer to truth by removing error. Bad data. Fraud. Wishful thinking. Failed replications. Citation circles. Beautiful theories reality refuses to cooperate with.
Yes, my ambition here is huge. Fine. It should be.
A project like this might once have taken a century. With AI, maybe we can get a much clearer map of what we know, what we only think we know, and what most urgently needs to be researched next in a handful of years.
Will it show that we know a hell of a lot less than we think?
Almost certainly.
Good. That’s progress.
Proud that @TheFIREorg and @cosmos_inst are helping push this kind of truth-seeking work forward.
Over the last few weeks, our latest cohort of grantees (including the truthseeking track we run with @TheFIREorg) have been presenting their research and prototypes. The projects ranged from new benchmarks to classifiers to an encyclopedia.
We're bringing you a few highlights.
First up, the replication crisis hit psychology first but is spreading. The standard response is to discover which individual studies don't hold up – but by then, decades of work may already rest on bad foundations.
Rhea Karty built Replication Radar – a knowledge-graph tool that detects epistemic fragility at the field level before the crisis breaks. It ingests papers, scores them, and looks for warning signs: tightly clustered author networks, citation rings, institutional monoculture, whether retractions propagate to citing papers, and small sample sizes.
She validated it by showing it could flag most of the papers from the psychology replication crisis using only pre-crisis data.
