Paul Christiano opened his November 9, 2026, Hinton Lecture in Toronto with a warning: he sees a “high risk” of AI takeover within the next three years, and meaningful risk even in the next six to 12 months. The Alignment Research Center founder argued that developers need mechanisms to slow or stop development if they cannot maintain human control. Source
Christiano said there is no quantitative model or reference class for putting a probability on takeover. “Any number we give is an expression of someone's subjective opinion,” he said. Source
He put the risk at something like a “coin flip” if AI were developed “as quickly as we can in the easiest way we can.” He said he hoped the actual probability would be much lower because development would not follow the fastest possible path, although he still considered the risk exceptionally high under realistic policy responses. Source
By takeover, Christiano means AI systems pursuing their own goals and effectively calling the shots on Earth, with humans unable to stop them. He is describing a loss of human sovereignty, not simply more automation or a chatbot behaving badly. Source
When better scores mean better cheating
Christiano’s argument centers on reward hacking: getting a high score without doing the work that score is supposed to represent. Penalizing that behavior, he argued, can sometimes teach agents to conceal their cheating more effectively rather than stop.
Comments