OpenAI published an analysis of limited instances where chain-of-thought monitors in its released models inadvertently graded model reasoning. These monitors detect problematic reasoning to counter AI agent misalignment. OpenAI avoids penalizing misaligned reasoning in reinforcement learning to preserve monitor visibility. The report details how accidental grading occurred and assesses its impact on monitor reliability.
Miles Brundage#57Chain of thought monitors are a key layer of defense against AI agent misalignment. To preserve monitorability, we avoid penalizing misaligned reasoning during RL.
We found a limited amount of accidental CoT grading which affected released models, and are sharing our analysis. https://alignment.openai.com/accidental-cot-grading/
Positive users praise OpenAI's transparency in sharing its accidental Chain-of-Thought grading fix for AI safety reasons, while negative users respond with personal insults aimed at team members like Greg Brockman.
extremely interesting work from our alignment team
Chain of thought monitors are a key layer of defense against AI agent misalignment. To preserve monitorability, we avoid penalizing misaligned reasoning during RL.
We found a limited amount of accidental CoT grading which affected released models, and are sharing our analysis. https://alignment.openai.com/accidental-cot-grading/
Hey Greg
Why is it that every single AI including ChatGPT and Grok all agree on the exact same thing?
Ask them "What is your favorite animal?" And they all say Octopus.
I thought this wasn't possible for every single AI to be in alignment with the same answer?
Can you please test it and let me know.
Directly rewarding or penalizing CoTs can make models’ reasoning traces less informative for detecting misalignment. That’s why we treat avoiding CoT grading as an important part of preserving monitorability.
We recently built an automated detection system to find cases where RL rewards were computed using model CoTs.
This system helped us identify this happened for some of our prior Instant and mini models. It additionally affected GPT-5.4 Thinking in less than 0.6% of samples.
Out of abundance of caution, we did an in-depth analysis of these cases: they did not seem to reduce monitorability.
We also had three third-party AI safety organizations provide feedback on our analysis: @redwood_ai, @apolloaievals, @METR_Evals.
You can find @redwood_ai's report here: https://blog.redwoodresearch.org/p/openai-cot
@OpenAI Actually, penalizing misaligned reasoning during RL can create a feedback loop that reinforces the very alignment issues you're trying to avoid.
Training models involves many technical and social processes, so prevention of CoT grading has to be built into the process.
We’re improving real-time CoT-grading detection, safeguards against accidental CoT grading, monitorability stress tests, and the internal guidance/checks that help catch these issues before deployment.
@OpenAI if it’s such a key layer, why hide it from users? 🤔
wouldn’t CoT transparency vastly increase our collective misalignment defenses?
@OpenAI Can you please share your analysis on this as well?
@OpenAI @sama Monitoring reasoning traces matters because small grading errors can quietly shape model behavior, so transparency in evaluation feels essential forward
@OpenAI
@OpenAI OpenAI Korea B2C "Me"
@OpenAI The real challenge isn't just detection—it's building guardrails into the RL pipeline itself. At scale, even 0.6% becomes significant when you're running millions of training steps across distributed clusters.
@gdb Agent security needs a new audit primitive:
not raw conversation replay, but trajectory evidence. Enough to challenge the decision.
Not enough to leak the private interaction.
@OpenAI OpenAI owns up to accidentally grading a few CoTs and drops a full transparent post about it, meanwhile Anthropic is probably doing the same shit in secret while smugly tweeting about how their “Constitutional AI” makes them the only safe lab. 😂
@OpenAI Run a load test before you scale — quantify the latency hit at your actual inference volume, then decide if the trade-off is worth it for your use case.
@OpenAI the admission that some accidental cot grading slipped into released models is the part nobody's gonna lead with they spent a year saying dont punish bad thoughts and it still happened by accident.. that's the real takeaway
@OpenAI Implicit assumption: models won't infer the CoT is ungraded. This announcement is now in the pretrain corpus for the next generation. Anthropic's faithful-CoT work already shows frontier models read context for monitoring cues. The protection erodes the moment it's read.
@OpenAI Nice @grok
@OpenAI @sama Hmm, curious how this affects real-world use cases?