Reactions from ranked influencers
2 posts
I'd like to do two things: (1) tell you where I think I erred in my original post about Kimi and (2) set the record straight about my views on open-weight AI. Before I joined OpenAI (and folks who are new followers: I joined the company less than two weeks ago and it is my first job in the technology industry), I often tweeted about the controversy of the day in a cold and analytic fashion. One aspect of my style was saying brutally honest things, including things inconvenient for my 'side' and my beliefs. The post about Kimi was largely written in that vein. When I said that the USG would probably realize its best option is to do ill-justified soft-law discouragement of Chinese AI, I wasn't proposing it like a good idea. Why on Earth would I do that? Why would I frame something I'm advocating for in such brutal terms? I am trying to describe what I believe will happen, not advocate for anything. My first mistake: What I now realize is that this style of analysis is no longer tenable. There is simply too much scrutiny on my words, too much temptation to draw conspiracies from my claims, and the like. It is my fault for not realizing this. Second mistake: I was relatively imprecise, writing, as I usually do, for a fairly high-context audience that was inclined to give me grace rather than pick apart every word. I should not have said, for instance, that open-weight models are unqualifiedly 'decelerationist'; I don't believe that. I only believe specifically that open-weight models decelerate capex spending on the margin, which is straightforwardly true. I did not say it was decelerationist for any other reason than that, and this is the only way in which I think open-weight AI is inherently decelerationist (though it is a big way). In many other ways, open-weight AI is profoundly accelerationist. Now, to where I stand on open-weight AI. My earliest experiences on the internet were posting in forums about philosophy, music, and movies in the early 2000s. Over time I realizes that forums on different websites had the same underlying forum software; this was the first time it occurred to me that 'software' is a thing people make (I realize this is a simple observation, but we are talking about an 11 year old with no prior exposure to computers). I looked into the software, and discovered that it was 'open source,' built and maintained for free by thousands of people around the world to facilitate the communication of millions of strangers. This notion struck me as deeply beautiful, and I decided I'd try to help. I began writing technical documentation, and later on, code, for this little forum project. This was how I first learned technical skills. I cherish that time, that software, and I have deep and abiding affinity for open-source software. The vast majority of the people commenting on my post have very little context for my prior writing. For instance, the fact that I wrote, in 2024, things like: "those who wish to hoard our software technologies may well be foreclosing on—or perhaps not even understand—the staggering civilizational victory that we earned through openness" or "I would like for AI to result in a similar smashing victory for America. To do that, we will need to set the global standard yet again. And to do that, we will almost certainly need to lead in open-source AI, because it is open protocols and open software that tend to define global standards in information technologies." I stand by these things. When I was in government, I worked alongside my colleagues to develop ideas and rhetoric that was strongly supportive of open-weight AI, and some of this work made it into the current US AI strategy. I stand by that work too. I also wrote, more than two years ago: "The day may come when frontier AI really is too dangerous to open source. If so, that will be a sad day. But we’re not there yet. Today’s models are not sufficiently useful—or dangerous—to justify such a drastic shift in public policy." I think it's pretty clear that we are approaching the point I describe--the point where, absent a major technical safety breakthrough, the national security implications of frontier open-weight model distribution are simply too severe. I don't think we're there yet (as I said in the piece), but the direction of travel is clear, and an analyst must be honest about this. Governments will realize these risks eventually, and when they do, they will have much lower risk tolerance than I have. We see this today with the Trump Administration, which once proudly championed open-source AI and now has a de facto licensing regime for frontier AI that I suspect will make it a challenge (if they still end up enforcing it) to release the weights of models of the "Mythos" tier. Every government will be safetyists once they understand themselves to be in the foxhole. You don't have to *like* this. I don't. But it is the reality as I see it, and what I have always tried to do with my writing is describe reality as I see it, even when it is inconvenient for me and my preferences. I intend to continue doing this. I will not be silenced by ignorant and loud critics. Yet I will have to work to find the new register I should adopt in my current job, which clearly changes the nature of my public communications even more than I had thought. But believe me: I'm not going anywhere. I am not retreating from public writing, and I am not retreating from saying inconvenient things in public. I am unfazed by harsh criticism, and I know that a reaction of this magnitude is in part the result of having struck a chord. Bear with me, and if you can, remember that I am a human being with a six-month old boy to raise, a book to write, a new job, and much more questions than answers about our collective future.
I’m afraid to tell you that it is effectively impossible to do the kind of writing I used to do on this website, not because anyone at OpenAI censors me but because of the sheer volume of hostility I get for sharing my analysis as a frontier lab employee. I enjoyed writing quick takes on this website for one basic reason: I could get rapid feedback on my own ideation process in real time. Post the early version of the take here, see the criticism; then refine, sharpen, and repeat. Unfortunately now that feature of this site is gone, because the feedback I get is now almost exclusively colored by resentment at the fact that I work at a frontier lab or other forms of hatred for my employer. The feedback signal is essentially useless now, so writing on here is not fruitful for me anymore. Literally everything I write now is responded to with “of course you said that because <OpenAI>.” I am truly just writing what I think and would have written anyway, but everyone reads what I say in the shrieking tone of “this is what openai thinks!!!!” (to be clear, my posts are not what openai thinks). This is an unpleasant and more importantly unproductive pattern for me. I anticipate that the shape of this account will change significantly as a result. I do not currently know how. It will not become a LinkedIn feed. It will change in some other way. It will no longer be a real-time accounting of my own thinking as it develops, since this is precisely the thing that seems impossible to do now. That will have to shift to private channels.
“The day may come when frontier Al really is too dangerous to open source” ‼️ Or too dangerous not to. Security through obscurity assumes a closed, elite group will find what the rest of the world cannot. Maybe, given enough eyeballs, all misalignment is shallow.
I'd like to do two things: (1) tell you where I think I erred in my original post about Kimi and (2) set the record straight about my views on open-weight AI. Before I joined OpenAI (and folks who are new followers: I joined the company less than two weeks ago and it is my first job in the technology industry), I often tweeted about the controversy of the day in a cold and analytic fashion. One aspect of my style was saying brutally honest things, including things inconvenient for my 'side' and my beliefs. The post about Kimi was largely written in that vein. When I said that the USG would probably realize its best option is to do ill-justified soft-law discouragement of Chinese AI, I wasn't proposing it like a good idea. Why on Earth would I do that? Why would I frame something I'm advocating for in such brutal terms? I am trying to describe what I believe will happen, not advocate for anything. My first mistake: What I now realize is that this style of analysis is no longer tenable. There is simply too much scrutiny on my words, too much temptation to draw conspiracies from my claims, and the like. It is my fault for not realizing this. Second mistake: I was relatively imprecise, writing, as I usually do, for a fairly high-context audience that was inclined to give me grace rather than pick apart every word. I should not have said, for instance, that open-weight models are unqualifiedly 'decelerationist'; I don't believe that. I only believe specifically that open-weight models decelerate capex spending on the margin, which is straightforwardly true. I did not say it was decelerationist for any other reason than that, and this is the only way in which I think open-weight AI is inherently decelerationist (though it is a big way). In many other ways, open-weight AI is profoundly accelerationist. Now, to where I stand on open-weight AI. My earliest experiences on the internet were posting in forums about philosophy, music, and movies in the early 2000s. Over time I realizes that forums on different websites had the same underlying forum software; this was the first time it occurred to me that 'software' is a thing people make (I realize this is a simple observation, but we are talking about an 11 year old with no prior exposure to computers). I looked into the software, and discovered that it was 'open source,' built and maintained for free by thousands of people around the world to facilitate the communication of millions of strangers. This notion struck me as deeply beautiful, and I decided I'd try to help. I began writing technical documentation, and later on, code, for this little forum project. This was how I first learned technical skills. I cherish that time, that software, and I have deep and abiding affinity for open-source software. The vast majority of the people commenting on my post have very little context for my prior writing. For instance, the fact that I wrote, in 2024, things like: "those who wish to hoard our software technologies may well be foreclosing on—or perhaps not even understand—the staggering civilizational victory that we earned through openness" or "I would like for AI to result in a similar smashing victory for America. To do that, we will need to set the global standard yet again. And to do that, we will almost certainly need to lead in open-source AI, because it is open protocols and open software that tend to define global standards in information technologies." I stand by these things. When I was in government, I worked alongside my colleagues to develop ideas and rhetoric that was strongly supportive of open-weight AI, and some of this work made it into the current US AI strategy. I stand by that work too. I also wrote, more than two years ago: "The day may come when frontier AI really is too dangerous to open source. If so, that will be a sad day. But we’re not there yet. Today’s models are not sufficiently useful—or dangerous—to justify such a drastic shift in public policy." I think it's pretty clear that we are approaching the point I describe--the point where, absent a major technical safety breakthrough, the national security implications of frontier open-weight model distribution are simply too severe. I don't think we're there yet (as I said in the piece), but the direction of travel is clear, and an analyst must be honest about this. Governments will realize these risks eventually, and when they do, they will have much lower risk tolerance than I have. We see this today with the Trump Administration, which once proudly championed open-source AI and now has a de facto licensing regime for frontier AI that I suspect will make it a challenge (if they still end up enforcing it) to release the weights of models of the "Mythos" tier. Every government will be safetyists once they understand themselves to be in the foxhole. You don't have to *like* this. I don't. But it is the reality as I see it, and what I have always tried to do with my writing is describe reality as I see it, even when it is inconvenient for me and my preferences. I intend to continue doing this. I will not be silenced by ignorant and loud critics. Yet I will have to work to find the new register I should adopt in my current job, which clearly changes the nature of my public communications even more than I had thought. But believe me: I'm not going anywhere. I am not retreating from public writing, and I am not retreating from saying inconvenient things in public. I am unfazed by harsh criticism, and I know that a reaction of this magnitude is in part the result of having struck a chord. Bear with me, and if you can, remember that I am a human being with a six-month old boy to raise, a book to write, a new job, and much more questions than answers about our collective future.