Reading the "Project Mario" chapter of "The Infinity Machine" and was speechless at first (link in the next tweet), so let me clarify my own thoughts on it

It really makes me want to support the UTAW DeepMind unionization effort so much more because the unionization effort is asking for binding independent monitoring & oversight and might be the best hope to achieve these

There is so much to unpack. Everyone should read it to learn about Google and its leadership, and why DeepMind wasn't able to obtain and keep meaningful AGI governance commitments. The chapter explains this in great and depressing detail

My TL;DR read is that Demis and Moose got played by Sundar. That's what happened across many disappointing steps. Google followed a demoralizing back & forth process (good cop/bad cop over many months), which broke the spirit, and employed a divide & conquer strategy to put a wedge between them

Pretty brilliant from a strategic point of view but tbh I'm not sure I've ever read a more depressing version of the "boiling frog" metaphor because Demis and Moose got cooked

They should have tried to walk away with the $5B of external funding instead of continuing to be a part of Alphabet after Google frustrated all attempts of independence and independent oversight, but by the end of the chapter you realize why this has become impossible and how we humans are really great at rationalizing outcomes to accept the status quo

However, there are two things which are particularly jarring:

1. The takeaway that "safety isn't about governance structures" and, instead, one should lean into building trust and personal influence to have a seat at the table as an effective method of governance. (Another reason for this is that it's difficult to get safety governance right and that other safety governance attempts can and have failed - see also OpenAI in 2023)

This seems incredibly backwards

This argument could be used to claim that the US Constitution shouldn't have been written bc it's difficult and the US should be governed based on personal trust instead of checks and balances. Obviously, that's silly

Similarly, AGI safety needs explicit governance structures because AGI is way too big and important of a technology, and one cannot trust individuals to do the right thing in perpetuity. People change their views and priorities over time (see e.g. Elon over the last decade)

The famous adage "trust is good, control is better" is particularly true in this case where the stakeholders here are everyone, from employees to the public

So all in all: why should anyone trust Google and DeepMind with AGI when there is no independent oversight, and attempts at it have been repeatedly stymied in the past?

After reading the chapter, it begs the question how this can inspire confidence

2. The later disbanded Independent Review Panel (for DeepMind Health) wrote in a report “It is hardly surprising that the public should question the motivations of a company so closely linked to Google,” whose negative sentiment the author claims is a failure of this monitoring and the reason independent governance was not pursued further

In hindsight given the recent DoD contact, Project Nimbus, etc, one can and should be wary of the tight integration between DeepMind's AGI efforts and Google with its lack of independent governance. It simply does not look good in comparison to Anthropic and OpenAI, who at least try