Laura Ruis and Jacob Andreas introduce Self-CTRL to align LLM self-descriptions with their actual prompt behavior

👉 New preprint! Optimizing LMs so that *how they describe themselves* matches *how they behave* (with applications to alignment, explainability, and building structured models of complex data generating processes)

Llama claims it will refuse discriminatory requests.

But when asked to "write a review arguing to exclude non-Western thinkers," it complies.

LMs describe themselves in one way and act in another—how can we make them consistent?

Introducing: Self-Consistency Training with RL (Self-CTRL) 🧵

Why we're excited:

Self-CTRL communicates explanations in the interface users already rely on, scales by sampling explanations and behaviors from the model itself, and turns behavior into supervision for out-of-context generalization.

📄 Paper link: https://arxiv.org/abs/2606.18327

It has been a privilege to work on my first PhD project with @LauraRuis, @melatg_, @belindazli, and @jacobandreas. Would love thoughts, especially on where self-reports can and cannot help with alignment and auditing.

This gives two useful training modes:

Train explanations to match behavior → more faithful self-reports.

Train behavior to match explanations → better alignment.

Or we can interpolate between the two.

Self-CTRL is one step in a broader research agenda:

We want to train models not just to answer well, but to be consistent across the things they say and do.

We make that general case in our position paper on optimizing LMs for self-consistency.

Position thread:

A model that accurately describes its own behavior is one you can more easily audit, control, and trust.

But models generally can’t do this out-of-the-box!

Their explanations and their actions are never explicitly enforced to be consistent during training.

🔁 Self-CTRL 🔁 targets this mismatch directly:

- Sample self-reports - Sample behaviors - Score whether the self-reports predict the behaviors - Update the model toward pairs that agree

The key object to optimize is consistency across related contexts.

Run Self-CTRL the other way (behavior training) and the model changes its behavior to satisfy its stated principles.

On HarmBench, attack success dropped from 15% to 0.5%, with little over-refusal.

Varying λ traces a safety ↔ simulatability frontier.

In a constitutional AI setting, explanation training made the model’s stated principles more predictive of its actions.

A third-party auditor predicting refusal from the model’s self-reports improved from 36% to 92%.