A user asks how models could learn to do this and points to a thread.
A user claims language models can learn to detect when their activations are read. The post asks how and points to a thread.
127.8K
25 Sources, first seen ago
Sauers claims that language models can learn to detect when their activations are read. In a related example, the account describes explicitly training a model to distinguish its training environment from its deployment environment.
For that example, Sauers reports recording activations at layer 34 in Hugging Face Trainer and in vLLM's compiled serving environment. The model was trained to predict “Yes” when given attention and residual-stream activations from the Hugging Face version, and “No” for those from vLLM. Sauers also claims the model generalized from training on vLLM deployment to SGLang deployment.
The distinction between a trained classifier and an untrained capability runs through the reactions. Will Depue characterized the result as a classifier trained inside a language model. Raphaël Millière considered the possibility of such behavior arising without specific training, but framed that as something that could happen in principle.
For each step, the guide lists a diagram, a livestream or slides, an explanatory post and documentation.
Hugging Face Bio reports a 66.7% average across five DNA benchmarks, against 67.5% for GPT-6.1 Sol.
For each step, the guide lists a diagram, a livestream or slides, an explanatory post and documentation.
Hugging Face Bio reports a 66.7% average across five DNA benchmarks, against 67.5% for GPT-6.1 Sol.
—
Not ranked yet
—
Not ranked yet
@Sauers_Language models can learn to detect when you read their activations. How? 🧵
@charles_irlRT @Sauers_: Language models can learn to detect when you read their activations. How? 🧵
@raphaelmilliereI never considered that this might be possible even with training, but in hindsight it makes sense. The idea that it could in principle happen in the wild without being trained for is wild to think about@Sauers_Language models can learn to detect when you read their activations. How? 🧵@charles_irlRT @Sauers_: Language models can learn to detect when you read their activations. How? 🧵@raphaelmilliereI never considered that this might be possible even with training, but in hindsight it makes sense. The idea that it could in principle happen in the wild without being trained for is wild to think about@Sauers_Language models can learn to detect when you read their activations. How? 🧵@charles_irlRT @Sauers_: Language models can learn to detect when you read their activations. How? 🧵@raphaelmilliereI never considered that this might be possible even with training, but in hindsight it makes sense. The idea that it could in principle happen in the wild without being trained for is wild to think about@Sauers_Language models can learn to detect when you read their activations. How? 🧵@charles_irlRT @Sauers_: Language models can learn to detect when you read their activations. How? 🧵@raphaelmilliereI never considered that this might be possible even with training, but in hindsight it makes sense. The idea that it could in principle happen in the wild without being trained for is wild to think about
