AI Researchers Risk Blindness To Social Consequences In AGI Pursuit
The AGI companies' best scientists are of course not the only stakeholders worth influencing. But any reasonable analysis acknowledges them as a particularly important set of stakeholders.
Seán Ó hÉigeartaigh@S_OhEigeartaigh"We were so involved in the technical problems, so fascinated by the extraordinary physics... that we suffered from a collective blindness to the social and political consequences of what we were doing." Victor Weisskopf on Los Alamos. Even though I'm not building it, I'm not immune to seductive awe and fascination at what is in the process of being achieved on the path to AGI and recursive self-improvement; especially now that AI is starting to push the boundaries of human abilities in mathematics, and stands ready to push the boundaries of other areas of science. It is a powerful and terrifying force; I think a far greater motivator for the best AI scientists than the (albeit incredible) money and status; and if we are to engage with this seriously, we will need to reckon with it (and, indeed, the costs of getting in its way) one way or another. I think it's a big weakness that so much of the community critical of this endeavour has settled on "it's all hype" and other cynical positions around this. Not just because (I think) it's likely wrong; but also because it means a majority of the criticism simply doesn't land with the people who are playing the biggest role in pushing and shaping this (many of whom whom are true believers). At least some part of the critical community needs to take it as seriously as they do, and grapple in good faith with the dark sides of the stakes they foresee.