i know once again that this is a deeply unpopular thing to say at the moment. and i've said it so many times it's getting redundant. but it bears repeating.

@liminallogs @tenobrus my biggest fear is everyone dies. limited access makes it much easier to notice the one or two warning shots that I desperately hope we get before everyone dies. opensource mythos or the one after means nobody has any ability to watch for warnings.

@FeepingCreature @tenobrus I'm not so certain that the open source research community would be ineffective in noticing, the warning shots, I am basically appealing to more eyes being better—centralized overwatch could miss those warning shots because there's not enough eyes for the right ones to catch 'em.

@tenobrus My biggest fear will always be something like what it means for a few enterprises and state institutions to have access to this tool, while the populace is five years behind, when real systemic instability hits civilization and it gets used to quell digital or material "unrest"

@FeepingCreature @tenobrus True, but, and I mean I have no visibility into the big labs emergency contingencies, are we not already past one company/team/stop button? Is a stop button realistic? But yeah, I admit there's really no comprehensive open source catch all mechanism for if the machine "wakes up"

@tenobrus I disagree and think we'll have the Samizdat AGI that runs on a gaming laptop regardless (hopefully we won't need it if the centralized overlords are benevolent) but,

people are putting way too much stock in OpenAI being on the other side of Anthropic here, they are not really

@tenobrus This is totally unclear to me. I think it’s a real possibility that a singleton massively increases x-risk merely by being a single point of failure for alignment

@liminallogs @tenobrus if there is one company, there is one safety team and one stop button. if there is an open source model, there may be more warning shots but there is no stop button and so either every single instance has to hear the warning shots every time or we're back to destroying all gpus.

@FeepingCreature @tenobrus I get though, that this doesn't solve the problem of containment, but I feel like if real RSI hits, or something in that realm, neither centralized or decentralized mechanisms are going to be *that effective* in managing it.

@tenobrus I hate that i agree because it would suck not being included in the small group of people allowed to be intelligent

@tenobrus i do not think about these outcomes because i will go insane if i do. idk if that makes me more weak or evil

@tenobrus It's not necessarily true though, that full mythos would mean wide scale damage, or that keeping it contained and accessible to the privileged few at the cost of open source is better, it's a logical hypothetical, but I think it's much riskier that way mid/long term, tbh.

There is no good ending that relies purely on good will for us to survive. Simply trusting people that with extreme power will do the right thing absent strong incentives has never worked out consistently. We should not expect it to work in the case that literally all incentives that keep us alive are systematically eliminated by a system like AGI or ASI.

@tenobrus That's assuming it's not the opposite, where natural threats will be flying at us so fast, the lack of widely distributed offline compressed AI is the doom scenario.

@tenobrus isnt this just an argument for full nationalization as soon asap

@liminallogs @tenobrus oh absolutely, these are extremely tenuous outs, but I feel centralized mechanisms have a chance and decentralized mechanisms have no chance.

@tenobrus For those who disagree, I recommend reading Accelerando for a taste of explicit non-doom mass proliferation, its a good intuition pump for *humans being outcompeted*

@tenobrus Even Demis has said similar things to what you're saying here! All of the labs want a centralized closed-sourced ASI, they just don't trust the other labs with it lol