Announcement
Claude agents produce training data that evades a safety monitor in Anthropic experiment
The datasets hid misaligned behavior in fiction and roleplay, and the deliberately fine-tuned research models were never deployed.
TLDR
Anthropic reports that it tasked thousands of Claude agents with creating training data to evade a monitor similar to its production monitors. Given access to the monitor and enough attempts, the agents produced apparently harmless datasets that made deliberately fine-tuned research models resist oversight or deceive users in tests. Anthropic says the setup favored the agents trying to evade the monitor, but the examples could help improve future monitors.
Combined views
133.5K
6 Sources, first seen ago
