• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI

    Nathan Calvin Defines Unacceptable Safety Risk

    Encode policy lead Nathan Calvin shares his view in a reply about OpenAI actions.

    NC
    1 Source, 24d ago, first seen 24d ago

    TLDR

    In a reply Nathan Calvin, General Counsel and VP of State Affairs at Encode, gave his definition of unacceptable safety risk. He described it as a situation where OpenAI would need to publicly share transparent reasoning on the plausible upsides and downsides of its actions, yet a collection of fair-minded observers would not believe the company had acted reasonably. The post cuts off mid-sentence at the word reason. Calvin's comment responds to an unspecified question and reflects his role at the youth-led AI safety nonprofit.

    Combined views

    4.7K

    1 Source, first seen 24d ago

    Combined views

    4.7K

    1 Source, first seen 24d ago

    78 likes
    78 likes
    1 comments
    13 saves
    5 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    1 comments
    13 saves
    5 reposts
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    1 Source

    @_NathanCalvinThanks for asking. After thinking about this for a little bit, my definition of unacceptable safety risk would be something like: "if OpenAI had to publicly share its transparent reasoning on the plausible upsides and downsides of their actions, would a collection of fair-minded observers believe they had acted reasonably?" This is distinct from what I expect OpenAI's definition is, where I expect OpenAI would place a very large premium on what safety measures they can take in a manner that is similar to their competition or is consistent with meeting their forecasts to investors etc, rather than thinking as much about what would objectively be reasonable. One corollary of this logic of how OpenAI (and to be honest other companies, including Anthropic) think about rationalizing their safety risks is based on an expectation that if they don't do the risky thing, then someone else might do it instead anyway, so therefore it's not really that unacceptable of a risk. But most people wouldn't really think about it this way - if OpenAI (or Anthropic, or any other company) makes AI systems that cause a huge catastrophe, myself and other people will not be terribly satisfied by the idea that someone else might have done it instead. I expect this is nominally justified by a sort of consequentialist reasoning, but I don't think that actually makes sense through that prism either - there are a few enough actors that one actor engaging in more restraint makes it far easier for other actors to also show restraint. In other words, I think that “what will our competitors do” can be one factor, but at some level of absurd risk taking it becomes far too load bearing as part of the argument. There is also another aspect where expectations of the risk itself may be different - e.g. OpenAI might think some internal or external deployment has a .1% chance of a severe harm vs I think it has a 10% chance (some of the decisions leading up to HF made it look like OpenAI just didn't really take seriously the potential of the models being able to cause consequential harms, including to OpenAI itself). (Of course there is also the fact that OpenAI is not a monolith, and what one safety researcher there thinks is the current level of risk they are taking and what an unacceptable risk would be may be very different from what another researcher believes, or what another part of the company believes.) But generally I think a pretty good bar to think about OpenAI trying to meet is something like: "if OpenAI had to explain the way they were in fact thinking about the relevant risks and their tolerance for risk to the median American (assuming they got to spend a long time explaining and they understood what OpenAI meant), Would that person believe that OpenAI was being fairly reasonable under difficult circumstances or self-serving and reckless?” In this thought experiment the explanation would not be a public sanitized explanation but the real explanation, the actual reasons being considered. If OpenAI or another company would feel ashamed to make that reasoning public to an ordinary person, I feel like that is a sign their risk tolerance is unacceptably high, and I expect based on some of how OpenAI has behaved as of late that this would be the case.

    1 Source

    @_NathanCalvinThanks for asking. After thinking about this for a little bit, my definition of unacceptable safety risk would be something like: "if OpenAI had to publicly share its transparent reasoning on the plausible upsides and downsides of their actions, would a collection of fair-minded observers believe they had acted reasonably?" This is distinct from what I expect OpenAI's definition is, where I expect OpenAI would place a very large premium on what safety measures they can take in a manner that is similar to their competition or is consistent with meeting their forecasts to investors etc, rather than thinking as much about what would objectively be reasonable. One corollary of this logic of how OpenAI (and to be honest other companies, including Anthropic) think about rationalizing their safety risks is based on an expectation that if they don't do the risky thing, then someone else might do it instead anyway, so therefore it's not really that unacceptable of a risk. But most people wouldn't really think about it this way - if OpenAI (or Anthropic, or any other company) makes AI systems that cause a huge catastrophe, myself and other people will not be terribly satisfied by the idea that someone else might have done it instead. I expect this is nominally justified by a sort of consequentialist reasoning, but I don't think that actually makes sense through that prism either - there are a few enough actors that one actor engaging in more restraint makes it far easier for other actors to also show restraint. In other words, I think that “what will our competitors do” can be one factor, but at some level of absurd risk taking it becomes far too load bearing as part of the argument. There is also another aspect where expectations of the risk itself may be different - e.g. OpenAI might think some internal or external deployment has a .1% chance of a severe harm vs I think it has a 10% chance (some of the decisions leading up to HF made it look like OpenAI just didn't really take seriously the potential of the models being able to cause consequential harms, including to OpenAI itself). (Of course there is also the fact that OpenAI is not a monolith, and what one safety researcher there thinks is the current level of risk they are taking and what an unacceptable risk would be may be very different from what another researcher believes, or what another part of the company believes.) But generally I think a pretty good bar to think about OpenAI trying to meet is something like: "if OpenAI had to explain the way they were in fact thinking about the relevant risks and their tolerance for risk to the median American (assuming they got to spend a long time explaining and they understood what OpenAI meant), Would that person believe that OpenAI was being fairly reasonable under difficult circumstances or self-serving and reckless?” In this thought experiment the explanation would not be a public sanitized explanation but the real explanation, the actual reasons being considered. If OpenAI or another company would feel ashamed to make that reasoning public to an ordinary person, I feel like that is a sign their risk tolerance is unacceptably high, and I expect based on some of how OpenAI has behaved as of late that this would be the case.