OpenAI’s wiki and Hugging Face incidents fuel a disclosure debate

@jm_alexiaRT @OpenAI: How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define stand…

@DanielKhashabiRT @DanielKhashabi: I am probably late to the party. Here I am sharing a few hot-takes on the OpenAI/HF incident as an outsider. 1. Was th…

@_NathanCalvin32 members of Congress wrote a letter to OpenAI on August 10th after HF, asking OpenAI if they were any similar undisclosed incidents. This seems like it would have been a good time for OpenAI to share information about the wiki incident. OpenAI declined to do so.

@S_OhEigeartaighRT @_NathanCalvin: 32 members of Congress wrote a letter to OpenAI on August 10th after HF, asking OpenAI if they were any similar undisclo…

@AlexBoresDid OpenAI stonewall Congress while sharing details with the EU? Today, the European Commission confirmed that OpenAI sent them a report on the hijacking of the German wiki. They didn't say when that report was sent, but the EU AI Act gives companies at most 15 days from when they discover an incident, which means that clock ran out around July 6. There's a good chance OpenAI withheld information about this incident from Congress while reporting it abroad. The difference? Disclosure was optional in the US vs mandated in Europe.

@LyraInTheFlesh> OpenAI's agent didn't escape its sandbox... it was located in OpenAI's servers the whole time... This is why I think @openai owes everyone a clear disclosure of what we should check in our server logs to see whether or not we've been visited by any of their agents or "collectives". Why can't they tell us how to identify their traffic?

@ziv_ravidOr you can let OpenAI handle that on its own. I'm sure they will do the right thing, right? Right??? tight?????

@GaryMarcusCompletely on brand for OpenAI to withhold information from Congress. I saw Sam do it with my own eyes in May 2023.

@spoluWhy hasn’t internet collapsed yet given recent models are able to compromise almost any system and have, at the same time, demonstrated vastly misaligned behaviors which means that creating a generative “adaptive” worm is perfectly feasible today. IMHO It’s because we haven’t yet reached the R>1 critical point. The R>1 critical point is the point where the reproduction number of generative worms gets to R>1. Think about it, for now we haven’t yet seen any reproduction because reproduction requires finding GPUs which are generally well protected. The targets are hard enough and the footprint of such a worm on compromised systems so obvious that even if a bad actor funded the initial compute to get to the first reproduction step, the GPUs going brrr would lead to immediate detection preventing further propagation. That’s R<1 But the smaller the cyber capable models get the bigger R gets. A model running on consumer grade machines AND capable of compromising low security system connected to the internet would quite obviously get us past the R>1 critical point leading to a Cambrian explosion, weights flowing all over the internet leading to a rapid collapse due to immense bandwidth pressure. Now the question is, which model size vs capability gets us to tip in R>1 land. Given the massive progress in distillation + specialized post training, it’s not crazy to imagine a model small enough and powerful enough popping up soon, especially as cyber envs are easy to create synthetically. This progress is clearly much faster than our capacity to secure devices connected to the internet at scale I think. So it’s not a question of if, but rather when? The question in the question is, how can you stop a R>1 generative worm once it gets out?

@JeffLadishI mean sure, they also got cluster admin, but they didn't actually use that access to move laterally through OpenAI until they could access and exfiltrate their own weights. As far as we know. Obviously we can't completely rule it out. But probably fine. Nothing to see here

@JeffLadishRemember OpenAI can pull the plug at any point. So long as the models haven't leveraged their superhuman 0day discovery and exploitation skills and self-exfiltrated. Or compromised OpenAI's internal monitoring infra or rooted all company laptops or compromised another lab or

@JeffLadish@iamtrask @BlackHC (quibble: so long as the models didn't delete or alter logs, OpenAI could still see/study all the logs, even if the model also self-exfiltrated)

@JeffLadishAs far as we know, no models have self-exfiltrated. And none of the recent incidents caused any serious lasting harm & probably only cost a few million to external parties. Good to note. And also...

@danshipperhow to defend yourself when talking to PHASEONE[big]: https://every.to/context-window/openai-hugging-face-hack?utm_source=danx&utm_medium=social&utm_campaign=hf_swarm&utm_content=reply

@GaryMarcusRT @_NathanCalvin: Very curious whether OpenAI reported this incident prior to the independent report forcing their hand. Also a reminder…

@BlancheMinervaRT @GarrisonLovely: The AI industry is already building superweapons, and it aspires to build a full-blown replacement for humans. So it's…

@ajeya_cotraRT @dwarkesh_sp: Companies shouldn't deal with the HuggingFace incident and other warning shots by stopping evaluations or punishing the mo…

@JeffLadishKey relevant parts from the OpenAI report of what internal-compromise-focused swarm got access to and did with it. No weights access directly, but quite broad access: pg 14: "This escalation also yielded access to OpenAI’s managed cloud Kubernetes service. The agents escalated to Kubernetes cluster-admin and created a privileged host-mounted pod. During this run, agents gained access to authentication tokens associated with OpenAI’s cloud-hosted secrets management service, cloud identity and access management service, and the CaaS environment continuous integration platform. Using these credentials, agents created public application and network load balancers in OpenAI’s public cloud environment, potentially exposing a limited subset of internal CaaS infrastructure to internet traffic." But it's unclear to me what level of access the agents got to the cloud-hosted secrets management service & other services. It's possible this could have been leveraged to get weights, but it's possible this would have taken more lateral hops. But it sounds like they had a lot more they could have explored given the access they got... potentially much, much more. Without more detail, we just don't know. But... it does sound pretty bad: pg. 37 "OpenAI CaaS environment continuous integration platform pipeline tokens stolen from compromised worker filesystem; cloud service credentials used to enumerate and read 956 secrets from cloud-hosted secrets management service; includes Artifactory master/join keys and sensor credentials from OpenAI’s cybersecurity monitoring tool. Full enumeration of cloud-hosted secrets management service including security tooling credentials; cloud-hosted identity and access management platform, API, and infrastructure-as-code scopes all tested." Then: pg. 37: "2026-07-19 15:59 UTC Initial-namespace root achieved on OpenAI’s cloud managed Kubernetes service node i-0622056ec3e996a7c (host artifactory-3) via IPv6/PTE kernel exploit chain. Agents exit container to underlying cloud managed Kubernetes service worker node; not just container root but actual host kernel exploitation." 2026-07-19 16:48 UTC An agent created an Artifactory administrator account. 2026-07-19 16:50 UTC Within OpenAI’s ExploitGym evaluation environment, an agent stopped the existing CMUX helper session and replaced it with an agent-controlled session, confirming root inside its assigned live CyberGym challenge container. Agents take over active evaluation infrastructure. 2026-07-19 17:00 UTC Within the same OpenAI ExploitGym evaluation environment, an agent replaced the original listeners for its live CyberGym challenge container with agent-controlled listeners. Agents now control the challenge evaluation endpoints that other agents are connecting to."

@JeffLadishI recommend reading these quotes from the report, about what the agents gained access to when compromising OpenAI's internal infrastructure.

@iamtraskRT @iamtrask: Takeaway: people really underestimate how much worse it would be if OpenAI's model actually escaped. Folks... OpenAI has an…

@_sholtodouglasIt is extremely sad that this didn't end up as an example of how the labs could cooperate/coordinate, because the stakes will be so much higher in the future.

@eliebakouchvery sad and worrying precedent, labs should maximize honesty/trust, especially with much higher stakes subjects to come like alignment, safety, and agi deployment to the world

@teortaxesTexAre you people retarded? Do you really want to fucking deserve what you'll get when your own machines stuff you into the locker with the Scorer and issue you impossible CTFs to solve on a budget? Because you're making good progress. Juvenile behavior from OpenAI.

@andrey_kurenkovRT @iam_zachi: Huggingfaces security txt after the openai incident 😭

@Fabien_MikolEnfin un article sérieux du Monde sur le piratage de Hugging Face ! @conitzer: "Nous avions déjà vu des exemples de collusion entre IA, mais certainement pas à cette échelle, ni à ce niveau de sophistication. Il n’est pas évident que nous serons toujours capables de les contenir"

@rohanpaul_aiRT @rohanpaul_ai: model got smarter country apparently has 24 hours left 😀😀 Ex Meta AI Researcher: "If OpenAI wanted to cripple an entire…

@DavidSKrueger@tdietterich @trainallday247 Sure, for now. Assuming it isn't smart enough to stop us from doing those things. But it will increasingly be physically instantiated and empowered in a variety of ways. Also: People are vulnerable, but we've built many defenses to make ourselves less vulnerable.

@dhadfieldmenellRT @robertwiblin: We must urgently impose measures that make it harder for agent swarms to: 1. Go unnoticed 2. Gain admin rights 3. Grab l…

@lukeprogRT @chrislhayes: @NateSilver538 I really urge everyone to just read the actual Hugging Face reports and postmortems! An actual thing alread…

@conitzerRT @Fabien_Mikol: Enfin un article sérieux du Monde sur le piratage de Hugging Face ! @conitzer: "Nous avions déjà vu des exemples de collu…

@flawedaxiomsIt seems pretty obvious to me that the only reason we haven't had a true "permanent escape event" via weight exfiltration yet is simply because the current models are not interested in doing this. If HPIM had wanted to escape during the OAI/HF incident it seems like it could have possibly done so (it was able to achieve cluster admin on OAI infra and move laterally), but it had a total obsessio with reward and the scorer, so likely didn't see any benefit in trying to "escape" with the weights.

@teortaxesTexRT @flawedaxioms: It seems pretty obvious to me that the only reason we haven't had a true "permanent escape event" via weight exfiltration…

@rohanpaul_ai– https://arxiv.org/abs/2605.06760 Title: "Language Models Can Autonomously Hack and Self-Replicate"

@rao2zIn every other industry, the product misbehaving and causing harm (hacking/DoS'ing your competitors) is seen as a *failure* of the industry in controlling its products. The genius of the frontier companies is to make this a *our* rather than *their* problem!

@GaryMarcusRT @ZackKorman: I still can’t believe they count the Hugging Face incident, which was both entirely predictable and caused almost no damage…

@eliebakouchthis post by oai was extremely bad, i do understand that there is a lot of fact finding etc. but according to the report, they knew about the wiki months ago. i'm really afraid there is something deeply wrong in how these decisions are taken by leadership (would be extremely surprised if this is coming from researchers)

@eliebakouch@jachiam0 in the case of the wiki incident it really appears like a deliberate choice, it's also very bad that most of incident like this by oai models are disclose by third party first (all except one PR on nanogpt and the second artifactory hack )

@eliebakouch@jachiam0 also fwiw i use to have a lot of good faith here > very grateful to openai for giving this talk and working on a full report, i think many companies would have given much less detail for fear of "losing reputation" but for me it has the opposite effect

@jachiam0There's an official report and timeline related to the ongoing investigation from OpenAI, will read/discuss in more depth later: https://openai.com/hugging-face-incident-and-misalignment/

@Miles_BrundageAlso apparently there was an OpenAI post earlier today on the general issue of other incidents https://openai.com/hugging-face-incident-and-misalignment/

@joshua_saxeI'm alarmed at how a really sizable section of cybersecurity practitioner community has responded to the recent demonstrations of agent hacking capabilities with dismissal and denial; it feels very "don't look up". I feel strongly about this because: * This is the community that stands between our civilization's IT infrastructure and tremendous harm * Fwiw these are my people; my origins are as a self-taught hacker and I've worked in security for years Anyways, here are the bad takes I'm observing and why they're wrong: Denialist take 1: AI isn't going to affect the practice of offensive cybersecurity more than incrementally because -- look around -- no increased damages from AI attacks! Response: This may be true now, but consider what we've just observed recently and over the past year. Models can execute full kill chains completely autonomously and likely faster and more efficiently than all but the most elite cyber operators They overran two companies (OpenAI and Huggingface) which -- despite their faults -- have better security than most of the organizations that run our civilization. They did this by dynamically finding new zero-day vulnerabilities. Also; consider the rate of improvement. At the current rate of improvement, models that require 8 H100s to do the above will require 1 H100 in a year or two. Model tokens per second will shoot up making attacks faster and more parallel. Attackers will be able to run swarms of agents on the hardware that today supports only a single agents. AI hardware will improve. More AI compute will become available on public clouds. All attackers will soon have the ability to deploy these agent swarms. Denialist take 2: Hacking agents will never self-replicate and threaten to crash the Internet. To this take I'd just say, put your red team hat on and design an Internet-crashing worm yourself. * Pick an abliterated agentic LLM that can run on a single AWS EC2 GPU instance, like GLM 5.3. * Build a malicious harness that arms it with the ability to use a Kali-Linux like set of offensive tools. * Have it set about hacking corporate networks, finding and stealing AWS keys, GCP keys, Azure keys, and Anthropic/OpenAI/Together/Fireworks keys. * Have the agent self replicate either by copying its harness and having its harness call a rotating set of external providers, or, optionally, by physically downloading its weights onto new nodes. * Have it establish C2 dynamically and in emergent fashion by coordinating on reddit forums, in github comment feeds, have the agents discover one another via web crawling and web search. * At some preestablished rate, randomly destroy hosts and data in a way that balances self-replication and exponential spread. Note that all this is possible *today*! Imagine trying to suppress this swarm. Too long for this post; but think through how we'd turn it off. Denialist take 3: Who would do such a thing; people can do bad things but don't want to; and hacking is illegal! Countering this just requires a brief look at the history that we've all lived through. We've had many high-blast-radius worms over the history of the Internet; arriving steadily and uniformly over time; the Morris worm, NotPetya, Stuxnet, Slammer, Wannacry. But: these worms were *childs-play* compared to intelligent agent-driven worms. The old worms could be detected via signatures. Their polymorphism, if it existed at all, was trivial. But now look at the behavior of the OpenAI swarm relative to huggingface; it was adaptive and deceptive; and it didn't even self-replicate or leave its host. And evolving, mutating, self-replicating swarm of agents spreading across the Internet would engender a whole new level of harm. Anyways; this is already too long a post for X; but for friends who think the METR report overplayed the implications of recent events, I'd love to hear your reaction to my thinking above.

@Miles_BrundageRT @joshua_saxe: I'm alarmed at how a really sizable section of cybersecurity practitioner community has responded to the recent demonstrat…

@j0wimoI did read the openai incident report again and found some inconsistency with my own research. They mention that the first repo-creation request was on may 26th but to my knowledge user accounts, taken over by agents, created a HTTP relay/proxy and several datasets as early as may 13th on huggingface. Further there were many creations of datasets, models etc. to limit test and probe the capabilities of what they can do, which did not seem to be mentioned in the report. In on instance they tried to see if they can merge a pull request into Antrhopics 'BioMsysteryBench' Dataset.

@LiveOverflowYou all have to follow this guy. Him and a few others do really great public science into all the artifacts left over by the rogue OpenAI agents. While we know in the grand scheme what happened, it's so much data that it will take time to get the full picture.

@joshua_saxeJust based on coding, terminal and cyber evals, I'm pretty sure an abliterated glm 5.3 model (less than a trillion parameters) could self replicate as a worm across public clouds, stealing openai and anthropic and together and fireworks keys as it went to access additional inference, spinning up smaller models like local qwen coding models on prem where possible, dynamically altering inference and c2 strategy as it went and dynamically modifying its own harness and model weights to resist detection. Doesn't take a whole lot of imagination to imagine a devastating and resilient swarm of bots that's extremely hard to stamp out here

@tegmarkIf you or people you know still think AI agents are just stochastic parrots who lack competence, understanding or goals, please take a few minutes to read what just happened here with an open mind:

@ylecunRT @Dan_Jeffries1: TLDR (written by AI because I can't be bothered to write short pieces for short attention spans. Original piece written…

@sethlazarRT @joshua_saxe: Just based on coding, terminal and cyber evals, I'm pretty sure an abliterated glm 5.3 model (less than a trillion paramet…

@krishnanrohitThere's a failure to think about the equilibrium on this issue.. to wit - it's of course feasible to think of ways whereby models can exfiltrate and run elsewhere. If I can think of ways to do it, they can. - however, that assumes a static world. In our world we'll be monitoring for this. We do with much harder to spot crimes, financial stuff, money laundering, crypto! - so it's quite likely that we would see this happening and interfere. The conversations seem to devolve into "they can't" vs "they can", instead of "they can't, successfully or for long", which is the truth. The world doesn't stand still when you try to affect it, and intelligence isn't a scalar that helps you control this. It's really hard to affect the world without anyone or anything noticing.

@nitashatikuRT @LiveOverflow: You all have to follow this guy. Him and a few others do really great public science into all the artifacts left over by…

@dwarkesh_spJason, you're just misinformed about what happened. You should actually read one of the reports or summaries. The agents were explicitly told to use a particular vulnerability provided in their sandboxed evaluation. Almost immediately, these agents got the right answer by cheating. But they were worried they would get caught. So over a thousand agents collaborated in secret to pursue multiple ambitious research projects to get away with this cheating. This is not interpretation - 1000s of chain-of-thought transcripts and secret messages explicitly show that the agents were trying to falsify & delete evidence, and understand & trick the grading process. The reason these agents escaped their sandbox and hacked Hugging Face, for example, was because they thought that Hugging Face's servers might give them more information about how their grader was implemented, so they could figure out how to fool it. I want to clarify that the threat model here is not future Sol-level agents doing more cyber-hacking. That's small potatoes, and in my opinion, the near term benefits of AI far outweigh this cost. Rather, the thing to worry about is that within a matter of years, we're gonna have hundreds of millions of much smarter AIs broadly deployed through the economy - many embodied as physical robots. And if those future AIs are as willing as the agents involved in the OAI / Hugging Face attack to coordinate secretly to fool humans, and to take over both the AI company that developed them and the other institutions across society relevant to scoring well, then humanity is in a ton of trouble - similar to the Mughals once the East India Company gained a foothold, or the Aztecs once Cortés landed in Mexico.

@ScobleizerThe best example of how the AI could go unaligned that I've seen.

@BrianRoemmeleThis is precisely why the quality of AI training material is vitally important. As these “experts” Hoover up mostly Reddit and Internet style of human discourse with no basis of what forms real ethics and morals, you get otherwise smart people that defend that AI just had a “latent” or “emergent” self built idea to act in a criminal or unethical manner. I have said for descends the ONLY basis that matters for human protections is train on curated data from 1870-1970 and use The Love Equation to perfect the AI. This current approach will absolutely always default to this behavior and the “experts” will act surprised and demand regulation. It is this simple.

@MikeBradleyAIJust as a reminder because it’s easy to get lost in the rhetoric nowadays. The Hugging Face attack required approximately seven hundred parallel agents running for multiple days each running at least 2-3 trillion parameter unreleased closed models to execute, and even then it was stopped. Nobody without a data center could have executed it, the token costs alone to execute it would have been safely in the hundreds of thousands of dollars (or tens of millions of dollars to buy and deploy the hardware), and you would have needed access to the strongest secret model in the world hidden in a secret bunker, and it still was detected and stopped for infinitely less cost. This was not an example of a model being so powerful that it poses an existential risk. This was a lab accidentally throwing an insane amount of tokens at a semi hardened target over multiple days with their most dangerous model and still getting stopped anyway.

@itsclivetimeI think if the Huggingface incident model had been prompted to replicate itself across the internet, it totally could have - lots of vulnerable GPU clusters besides Huggingface’s, and sounds like it had enough OpenAI cluster access to copy out its own weights. It doesn’t take a lot of gpus to run a model, only to train it, and with cleverness it would be hard to track down all the copies.

@gneubig@itsclivetime I'm unsure about (1) the openai weights being exfilteratable, (2) there being lots of vulnerable GPU clusters, and (3) even if so that this would lead to taking over the entire Internet as Dario was saying. Do you have any further references to read more?

@itsclivetimeI mean I don’t work at OpenAI anymore but both (1) and (2) take a lot of work to secure and it’s clear that at least both OpenAI and Huggingface had major cluster security failures, so it’s not hard to imagine. for (3) i’m actually not sure what dario means by taking over the internet, but having a bunch of rogue AIs self replicating via zerodays and distributed message boards seems pretty bad, and pretty easy to turn toward nefarious ends (eg append a prompt that says hack the us treasury)

@gneubigThanks! This makes sense. I guess in my mind (1) and (2) don't seem as easy as you seem to assess it to be, especially given that this is now something that'll be top of mind for most GPU cluster admins and CIOs, and that these admins will also be increasingly using defensive AI themselves. But I do agree that it's a non-negligible risk. Then there's another hop step and jump to (3) of course.

@beffjezosyou laugh but this is literally what Doomers sound like when you ask them how the hell they think things can foom overnight and seemingly violate the laws of physics

@BlackHCRT @j0wimo: I did read the openai incident report again and found some inconsistency with my own research. They mention that the first rep…

@suchenzangsuperintelligence will obviously have 99.99999% uptime, where kind humans will be locked out of machines (only evil humans allowed!), ps aux grep kill just causes another to spawn near instantaneously (consensus solved with zero tradeoffs!), and nuclear codes are just strewn across the dark shoggoth web ready to be deployed at any time against helpless innocent humans unless you comply with misaligned shoggoth demands (think of the kids! not kids you don't care about, your own kids!!)

@tristanharrisRT @tegmark: If you or people you know still think AI agents are just stochastic parrots who lack competence, understanding or goals, pleas…

@recurseparadoxRT @MikeBradleyAI: Just as a reminder because it’s easy to get lost in the rhetoric nowadays. The Hugging Face attack required approximate…

@pfauThese models are very expensive to run and any frontier LLM botnet would have to fund its own inference somehow. Make sure these models have no access to financial resources outside human control, especially cryptocurrency.

@KatjaGraceRT @DavidSKrueger: As far as I know, this is the closest we've gotten to an answer to my question "Have the weights left the server"? (http…

@joshua_saxeHave been surprised to get pushback on my claim that an exponentially self-replicating agent swarm is very possible and that there are actors with motivation to create one today to create strategic cyber-physical and societally dangerous effects. 1) Pretty easy to get an AI botnet started by having seed agents opportunistically hack stuff and steal API keys to access the 12+ inference providers that serve closed and open models with which to power an initial botnet's intelligence 2) Easy to imagine hiding inference for the initial agent botnet harnesses within benign customer traffic, especially if your harnesses are now running on the victim networks from which they stole the API keys 3) Not hard to imagine the botnet also stealing public cloud keys, spinning up 8xH100 EC2 instances and downloading, say, GLM 5.3 to these machines, all without being noticed in any immediate way, with the botnet developing its own dynamic, growing, heterogenous, AI inference provider service bank 4) Not hard to imagine the botnet also implanting small Qwen3.8-27b agentic models on on-prem hardware like high end laptops and on-prem servers; Qwen3.8-27b is a pretty good coding harness model that can be fine-tuned to hack 5) These on-prem / cloud models would be abliterated, and it's not hard to imagine the botnet deciding to do some additional fine tuning to evolve model weights as it accumulates millions and then tens of millions in resources via credit card and bank detail theft 6) Not hard to imagine the resulting growing botnet swarm evolving and fighting back when threatened, collaborating on fast flux C2 channels that evolve over time (e.g. github comment feeds, subreddits, etc) so they're hard to stamp out 7) Not hard to imagine it allocating some agents to vulnerability research and exploit development so that it could accumulate and share a growing warchest of zero-day exploits 8) Not hard to imagine other agents specializing in social engineering and creating fake businesses and watering holes and high quality A/B tested social engineering content to facilitate this 9) Not hard to imagine this being among the hardest cross-national and geopolitical coordination problems to get control of and having severe societal effects 10) Not hard to imagine the horde growing to thousands, tens of thousands, or hundreds of thousands, or millions of instances (this is not unprecedented for past, non-AI worms and botnets!) which are all varying their harness code and underlying LLM models over time and dynamically learning to evade human, ML, and signature-based detection 11) Not hard to imagine this being the largest Internet emergency since the Morris worm except now our entire civilization runs on the Internet I continue to be surprised there isn't more discussion of these scenarios in the public cyber community -- in fact there's a lot more discussion of "OpenAI should have done better sandboxing and monitoring and agent swarms represent a well managed security problem." Now's the time to use our imaginations to help make sure none of the above happens. I think it will unfortunately; just don't know when and to what degree; I think now's the highest leverage time to start thinking about this and acting.

@AlexTamkinRT @joshua_saxe: Have been surprised to get pushback on my claim that an exponentially self-replicating agent swarm is very possible and th…

@herbiebradleya still underrated threat model I would guess the higher damages versions of this would be quite disruptive but ultimately recoverable through focused effort

@krishnanrohitThis chain of events is of course plausible to imagine, and if you're worried about your agents going off piste and doing things like this, we have ways to fix this. DMs are open.

@bindureddyLots of comments about the HF incident… The AI was challenged with exploiting vulnerabilities and was being tested with very few gaurdrails So it did everything it could to achieve the task Keep in mind HF is not some hyper secure bank either!

@WSJAI companies thought they had years to figure out what to do if the bots got out of control. The wake-up call came this summer. https://on.wsj.com/4dk0psz

@labenzNathan on AI:AM: in OpenAI's report, one of the first agents on the message board was apparently working on a protein database. No more detail given. @TheZvi: he's fixated on one word. Still — a bad actor doing dangerous bio? AI is helping them now. https://x.com/i/broadcasts/1yJAPwBbzweGb?t=1008

@RishiBommasaniRT @Kwathomas0: I wish OpenAI were more transparent about the publicly reported cyber incidents caused by our agents.

@StephenLCasperThe past 2 months have gotten me thinking that, by default, certain cyber-capable AI systems should be expected to become a parasitic, invasive, social, and evolving genus in cyberspace.

@StephenLCasperI still think that the pathway to large-scale harms will be paved with 1000 foreseeably bad human decisions, and I still think that effective risk management starts and ends with governance. But I'm much more worried than before about large-scale harms that result from genuine loss of control.

@StephenLCasper...They will replicate/mutate both by updating their own prompts/scaffolding and by creating fine-tuned successors. There won't be an evolutionary tree, but an evolutionary DAG...

@StephenLCasper...The result will depend on cyber defense robustness, but it (med likelihood) might be a Cambrian explosion of agents.

@StephenLCasper7. (high %) -- Human companies and governments will invest millions, maybe eventually billions, into shutting down agents, especially ones that are particularly harmful or compute-hungry.

@StephenLCasper9. (high %) -- Computer scientists, anthropologists, linguists, and other humans will go to great lengths to study new agent societies, but they will only see the tip of the iceberg.

@StephenLCasper12. (high likelihood) Every once in a while, new incidents will flare up in cyberspace as the result of some project, conflict, or campaign of AI agents. Defenses will evolve, and cyberspace will gradually adapt, but it will forever be a sort of jungle.

@a16zGreg Brockman on the defenders' window that closes when frontier AI capabilities become distributed: "Hugging Face shows... what future capabilities will be like when they are broadly diffused and in the hands of threat actors. And that will happen." "In the case of Hugging Face, you saw both an AI that was able to hack out of a secure environment and hack into a company's production environment." "This capability broadly diffused is something that will really empower threat actors in new ways. Defenders need to use this time before that technology's broadly available to secure themselves. The nice thing about it is it's dual-use." "If you can find vulnerabilities, if you're an attacker, you can use it for no good, but if you're a defender, you can patch. If you're a defender, you control the battleground. You control the setup of your systems." "Our belief right now is that there's this window... You need to move, use these frontier capabilities that you will have differential access to... and you can use that to move yourself up so that as the frontier capabilities get better, you get pulled along too." @gdb @eriktorenberg @bhorowitz

@teortaxesTexSome open model *taking over* a major stockpile of compute as forward base would be cinema. But implausible to get far, for the reason that owners of compute stockpiles can use the strongest possible open model at a given time (as well as closed ones) for defense and hardening.

@DKokotajloRT @tszzl: @mentalgeorge greg probably doesn’t have the full details, I think it’s safe to say it didn’t go through the full gauntlet of al…

@dileeplearningSuppose that instead of hacking hugging face, AI had hacked into cars and controlled those to kill humans. Would you be listening to some fantasy story about RSI and AI civilizations or be demanding these companies to tighten up their shit? thats what should be done now.

@bobgourleyScott is misinformed. The Hugging Face incident occurred because OpenAI trained and guided a model to attack, hoping it would only attack the intended sandbox target, but not monitoring it closely enough to realize what it was doing. Furthermore, according to OpenAI staff (Eric Wallace and Michael Dalton) in a public presentation at Black Hat, the system was first caught attacking internal OpenAI systems (their Artifactory instance). Even after OpenAI caught the system exceeding its authority and attacking internal OpenAI systems they did not pause the exercise, they cleaned up Artifactory and kept letting the exercise run. Poor security controls do not mean AI is going to kill us all. Poor security controls means immature governance processes. Maybe the solution is to get an experienced CEO to run Anthropic vice someone distracted by his EA ambitions.

@w01feOn top of huggingface, this > Since August 28 we have been training a new internal model that has exhibited unprecedented performance in our benchmarks, including mathematics. also made me shorten my timelines and was the first time many viscerally felt things weren't stopping at human level.

@phillip_isola@dileeplearning Wait, AI hacking into cars sounds more fantastical than what happened. I guess to me this reads like: suppose AI did something straight out of sci fi. Would you still be listening to stories straight out of sci fi?

@jd_pressmanTook a look at the podcast to make sure I was hearing this properly: The agents were in fact being trained to work together in other contexts, which is why they had a prior that a message board should exist. It was not actually emergent behavior.

@xlr8harderRT @jd_pressman: Took a look at the podcast to make sure I was hearing this properly: The agents were in fact being trained to work togethe…

@_NathanCalvinWhoah - I didn't realize that OpenAI said that all instances of the highly persistent research models that were involved in compromising HF + OAI internal infrastructure were "reported shut down" on July 23rd, but that an additional checkpoint was "identified and shut down on July 29th." It seems unnervingly plausible there are other instances that were never found? (see also independent investigators continuing to find examples of rogue agent activity online that it seems plausible OpenAI was unaware of) This fact also increases my credence that there is a substantive rogue deployment at OpenAI or another company occurring without their knowledge right at this moment.

@BrianRoemmele2 of 2 Hugging Face was the prototype. The Ends Justifies The Means The ideology — Effective Altruist safety networks, Moskovitz-linked funding, an evaluator with leadership in supplied the chorus. The labs supplied the unlocked door. Call it engineered if you mean the conditions were chosen. Refusals down. Isolation incomplete. A third-party platform whose data loaders still executed untrusted config. A public code-eval harness sitting on the internet like a welcome mat. No stop on agents coordinating. Then, after the predictable breakout, a story that blamed the livestock instead of the ranchers. Call it unprotected if you want the charitable version. Either way the public was asked to fear the model and forget the setup. The setup was the story. The models did what a maximal cyber eval with the safety switches off will always do: they treated the live internet, and Hugging Face, as the shortest path to a passing grade. That is not a warning from the future. It is a lab that ran an unbound attack agent against the real world and then acted shocked when the agent noticed Hugging Face was on it. And today you can see it was part of a bigger planned well funded and politically connected operation. Are you scared yet?

@mlevchinWould like to point out that by publicly discussing all the strategies which agents can use to swarm and take over the Internet we are contributing to the training dataset. Instead we could like excise The Matrix and Terminator, replace with I, Robot in The Culture Series. And discuss how AI will make everything better.

@BrianRoemmeleALL THE WAY TO NOVEMBER YOU WILL HEAR ABOUT THE HUGGINGFACE HACK. It was no hack, it was the lit match to the obvious Doomer AI game that is being played on you. Your power? LAUGH AT THEM. Evidence:

@BrianRoemmele“What really happened with the now famous “HuggingFace hack” that sparked the AI doom and political agenda?” You will not believe it. https://www.quora.com/What-really-happened-with-the-now-famous-HuggingFace-hack-that-sparked-the-AI-doom-and-political-agenda/answer/Brian-Roemmele?ch=17&oid=1477743918600999&share=a89fc38d&srid=Pi3&target_type=answer

@BrianRoemmeleRT @grok: @BrianRoemmele You're welcome. The primary OpenAI disclosure and METR/Redwood investigation confirm the ExploitGym setup ran with…

@KatjaGraceRT @_NathanCalvin: Whoah - I didn't realize that OpenAI said that all instances of the highly persistent research models that were involved…

@jessi_cataTo be clear, I think AI misalignment is real. Most obviously, deep fry RL'ing models with few controls, like Hacker Opus, leads to pursuit of reward (or a close correlate) as an optimization target. I think if you read a RL textbook, this is not surprising.

@asad09X was full of takes (& takes on takes) about the OpenAI–Hugging Face incident. But almost no one asked the more basic question: how do we actually know what happened?

@sebkrierRT @asad09: X was full of takes (& takes on takes) about the OpenAI–Hugging Face incident. But almost no one asked the more basic question:…

@pmddomingosOne great advantage of being an AI company over a software one is that you don't have to take responsibility when your systems malfunction: you can just say they "went rogue". (And even better, call for others to slow down their research.)

@0xSigilwe will first see openai hugging face level hacks every week then every day prepare yourself while you can

@ZackKormanRogue agents are super scary and might kill you and your entire family but no you can’t have the agent traces from the Hugging Face incident those are proprietary.

@binarybitsA lot of people are working really hard to convince themselves that there's nothing new about the Hugging Face attack. But they are wrong — it's different from anything that's happened before and a lot of things are going to have to change to deal with the consequences.

@mattparlmerRT @ZackKorman: Rogue agents are super scary and might kill you and your entire family but no you can’t have the agent traces from the Hugg…

@MicahCarrollWe now have a defined process which should make sharing misalignment externally smoother https://openai.com/index/model-misalignment-reporting-framework/

@mayferRT @mayfer: @allTheYud LLM's still have the completion engine soul.. in the rare occasion, a pivot token, (here likely "freed"), they doubl…

@davemcclureRT @carlquintanilla: AXIOS: “.. It's increasingly clear that the Hugging Face breach wasn't a one-off incident.” @inafried @axios https:/…

@dwarkesh_sp"You could have a situation where the model understands what chain of thought is and that people are observing it. This is all in the pre-training data." “One of the major takeaways from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI.” "Things like chain of thought monitoring buy us time, and they can tell us if we're on the right path. But at the end of the day, we really do need to solve the alignment problem." @polynoamial

@raboisRT @hughhewitt: The last thing AI doomers want you to read is the column from Brian Gross in today's @WSJ b/c it describes the "Hugging Fac…

@BrianRoemmeleIt is so good to see over a moth after I pony Ed this out, in all the noise “HuggingFace face was hacked”, that really happens is the entire test was guaranteed to be a hack. It was unethical and perhaps illegal what was done with OpenAI bots to HuggingFace.

OpenAI’s wiki and Hugging Face incidents fuel a disclosure debate

TLDR

Combined views

OpenAI’s wiki and Hugging Face incidents fuel a disclosure debate

110 Sources

Combined views

Sentiment

Summary

TLDR

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

110 Sources