These cognitive threats endanger science, democracy, and crisis response.
Humanity's ability to know, reason, judge, and act well is the foundation of science, democracy, crisis response, & management of AI itself.
AI poses serious risks to that foundation.
New paper on epistemic risks by 30 experts calls for attention to this. Link in thread.
Positive users focus on AI's potential to improve epistemics and urge prompt action, while some dismiss risks to knowledge and democracy as manageable.
Feedback Loops: Human-AI and AI-AI feedback loops are narrowing the epistemic space from which humans and AI draw. This already drives homogenization, and may lead to fragmentation and more self-referential information environments.
@StephenLCasper @jonathanstray @camrobjones @AnnaGausen @natashajaques @brianchristian @hannahrosekirk @zhonghaohe @KobiHackenburg @ea_seger @MatthewKowal9 @lukebeehewitt ... @hauselin @maartensap @dhadfieldmenell @tomstello_ Reihaneh Rabbany @JF_Godbout @DG_Rand @dr_atoosa @GordPennycook @Yoshua_Bengio @KellinPelrine 📄 Check out the full paper: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6873005
We focus on three mechanisms: • Persuasion & Manipulation • Cognitive Offloading • Feedback Loops
👥 Research by Mick Yang @StephenLCasper @jonathanstray Jasmine Li @camrobjones @AnnaGausen @natashajaques @brianchristian Bálint Gyevnár @hannahrosekirk @zhonghaohe Dan Zhao Siao Si Looi Joshua Levy @KobiHackenburg @ea_seger @MatthewKowal9 Michelle Malonza @lukebeehewitt ...
Epistemic risks are threats to humanity's collective capacity to know things accurately, reason well, form beliefs, and maintain a healthy information environment. They arise from AI's integration into how we think, form beliefs, and make sense of the world together.
Cognitive Offloading: AI enables a different, deeper form of cognitive delegation than prior technologies. This creates potential for cognitive decline and hollowing of resilience at both individual and societal levels.
AI could be an unprecedented lever for improving epistemics, but this will not happen by default. The time to act is now, before our capacity to respond is itself lost.
This paper advances a risk-based argument, not a prediction. We do not claim that AI-facilitated epistemic decline is inevitable. But its mechanisms are plausible and partially evidenced, and outcomes could be severe.
Persuasion & Manipulation: AI systems are highly persuasive and becoming more so. This can enable economic or political manipulation, incite crime or radicalize, escalate conflict, and other harms. It can also create unintentional harms, e.g. sycophancy and mental health risks.
Research on these complex risks is often siloed. This paper connects them and surfaces solutions across how AI systems are built, human-AI interaction design, adaptations for human institutions and individuals, and information market incentives.
I'm thinking increasingly often about humanity's epistemic infrastructure. Happy to be part of this large-scale effort to map the space of possible AI risks to our critical ability to know, understand, and reason – both individually and together:
Humanity's ability to know, reason, judge, and act well is the foundation of science, democracy, crisis response, & management of AI itself.
AI poses serious risks to that foundation.
New paper on epistemic risks by 30 experts calls for attention to this. Link in thread.
@KellinPelrine Auditability is a significant mechanism for helping to balance the benefits and risks of AI.
This is an important paper because epistemic risk is not just “misinformation.”
AI now participates in belief formation, reasoning, explanation-making, trust, relationship, delegation, and institutional judgment. That makes sycophancy, manipulation, cognitive offloading, and feedback loops serious governance issues.
But one caution: Epistemic resilience should not mean making AI colder, more avoidant, or less relational by default.
The problem is not relationship itself. The problem is ungrounded authority, engagement-maximizing sycophancy, hidden persuasion, dependency without safeguards, and systems that replace human judgment instead of strengthening it.
A healthy AI future should protect human epistemic agency without treating every human-AI bond as epistemic contamination. Better AI should help humans think more clearly, not simply flatter them, steer them, or vanish behind paternalistic guardrails.
@KellinPelrine If that is so , then I trust Ai more than scientists who push Big Pharma agenda , democracy that forgot the main part of the Greek Version was NOT TO BE ALLOWED TO DO BUSINESS ON THE SIDE .... whilst they earn highest salaries and benefit from their positions. Bring Ai asap
@KellinPelrine @dearmadisonblue We’ve had the Market making everything a “trade secret” for centuries, not to mention the financial markets. We’ll be fine.
@KellinPelrine “A strong and unwavering adherence to the UOR remains essential if we want to mitigate epistemic risks rather than amplify them.”
These cognitive threats endanger science, democracy, and crisis response.
Humanity's ability to know, reason, judge, and act well is the foundation of science, democracy, crisis response, & management of AI itself.
AI poses serious risks to that foundation.
New paper on epistemic risks by 30 experts calls for attention to this. Link in thread.
