Richard Ngo and Toby Ord examine how AI safety efforts produce unintended outcomes including accelerated progress and institutional advantages from safety rhetoric

Part of the problem is conflating actual attempts to understand things with the use of “understanding” or “safety” as fig leaves for power-seeking.

Rationalists do the former much more than EAs, which is why they have many fewer prestigious jobs, lab equity, etc (and get called weird much more). Most of the best thinkers I know (e.g. people I follow on @researchngo) are institutionally independent by now.

In general my advice is to cultivate virtues, many of which are culturally transmitted defense mechanisms against counterproductive behavior. Virtue is hard to pin down but it’s often fairly obvious if you’re looking for it. As one example, the old Racing to the Precipice paper had much more intellectual virtue than Situational Awareness—the latter was skewed in many ways by social incentives, as you can see by reading the text.

Meanwhile most “safety” work inside labs is so shaped by corporate and/or ideological incentives that it’s hard to extract meaningful signal from it. Few of these papers are good science, but they feel so useful (slash people are so panicked about short timelines) that the field embraces them anyway.

More generally, these days it’s much easier to convey important ideas with nuance and clarity in blog posts than in papers. Insofar as people still write papers for the prestige and recognition, I’d say that’s also less intellectually virtuous. (There’s still a role for papers in pinning down a legible idea for adversarial scrutiny, but I’d recommend something like a 10:1 ratio of posts to papers on a given topic.)

Part of the problem is conflating actual attempts to understand things with the use of “understanding” or “safety” as fig leaves for power-seeking.

Rationalists do the former much more than EAs, which is why they have many fewer prestigious jobs, lab equity, etc (and get called weird much more). Most of the best thinkers I know (e.g. people I follow on @researchngo) are institutionally independent by now.

In general my advice is to cultivate virtues, many of which are culturally transmitted defense mechanisms against counterproductive behavior. Virtue is hard to pin down but it’s often fairly obvious if you’re looking for it. As one example, the old Racing to the Precipice paper had much more intellectual virtue than Situational Awareness—the latter was skewed in many ways by social incentives, as you can see by reading the text.

Meanwhile most “safety” work inside labs is so shaped by corporate and/or ideological incentives that it’s hard to extract meaningful signal from it. These papers are not good science, but they feel so useful (slash people are so panicked about short timelines) that the field embraces them anyway.

More generally, these days it’s much easier to convey important ideas with nuance and clarity in blog posts than in papers. Insofar as people still write papers for the prestige and recognition, I’d say that’s also less intellectually virtuous. (There’s still a role for papers in pinning down a legible idea for adversarial scrutiny, but I’d recommend something like a 10:1 ratio of posts to papers on a given topic.)