Google DeepMind's interpretability team tested pretraining checkpoints of Gemini 3.1 Pro and Flash by running them through the standard supervised fine-tuning mix and found the resulting models already displayed most of the safety properties seen in the finished versions, shifting attention away from reinforcement learning as the main alignment driver.
Gemini 3.1 Pro and Gemini 3 Flash have most qualitative behaviors set by SFT, not RL, contrary to my expectations!
New GDM interp research: SFT is a big deal for safety relevant behaviors.
We recently investigated root causes for some of Gemini’s behaviors. We were surprised to find that many behaviors actually came from the initial supervised finetuning stage, not later stages like RL!
🧵
The post-SFT checkpoints matched production safety benchmarks closely enough that later RL stages appeared secondary, a result that overturned the team's prior expectations and points to supervised fine-tuning as a high-leverage moment for shaping model conduct.
Exact behaviors measured, quantitative gaps, and comparisons to other models stay unspecified in the current update, leaving open how broadly the pattern holds and what targeted SFT changes the team will try next.
Many users praised the intellectual honesty in the realization that SFT shapes Gemini safety behaviors more than RL and called the insight super interesting, while one dismissed RL work as merely fixing SFT issues.
At the start of this project I assumed RL was the source of basically all misalignment - I was pretty surprised to be wrong! I think these results will plausibly change over time, but SFT still seems likely to be important for aligning frontier models
New GDM interp research: SFT is a big deal for safety relevant behaviors.
We recently investigated root causes for some of Gemini’s behaviors. We were surprised to find that many behaviors actually came from the initial supervised finetuning stage, not later stages like RL!
🧵
New GDM interp research: SFT is a big deal for safety relevant behaviors.
We recently investigated root causes for some of Gemini’s behaviors. We were surprised to find that many behaviors actually came from the initial supervised finetuning stage, not later stages like RL!
🧵
At the start of this project I assumed that to fix misalignment we mainly needed to intervene on the RL stage of training, and SFT didn't matter much - I was pretty surprised to be wrong!
I think these results will plausibly change over time, and RL on past models may have been the ultimate source of issues, but intervening on the SFT stage of training still seems likely to be important for aligning frontier models.
New GDM interp research: SFT is a big deal for safety relevant behaviors.
We recently investigated root causes for some of Gemini’s behaviors. We were surprised to find that many behaviors actually came from the initial supervised finetuning stage, not later stages like RL!
🧵
Post here: https://www.alignmentforum.org/posts/nLrrYweeFxgXACSmS/sft-drives-gemini-s-safety-properties-1 Work done with @ArthurConmy, @bilalchughtai_, and @NeelNanda5.
My intuition for what’s going on: when we distill from a set of rollouts in SFT, we are establishing a strong prior on a certain assistant persona. Traits of that persona mostly aren’t explicitly optimized against in later training stages like RL and so don’t change that much.
This work is part of a series of research updates from our team:
We do not want to overstate this claim as applying to other model families, and we also note that our results may change in future Geminis. Nevertheless, this result was counter to our expectations, so we felt that it was important to share with the broader safety community.
This work will inform future safety work on our team. For example, one implication of this result is that one should prioritize intervening in SFT for making robust downstream changes to model behavior.
@NeelNanda5 that’s a wild insight, what made you change your mind?
@NeelNanda5 that is a super interesting realization, how did you find out?
@NeelNanda5 @slimer48484 sft does matter huh. all that rl work just fixing sft mess.
@NeelNanda5 love the intellectual honesty in admitting you were wrong publicly
this part of the field needs more of that energy
@NeelNanda5 SFT being the hidden lever is the kind of thing you only catch after spending months debugging a pipeline.
@NeelNanda5 the RL assumption was so widespread it was almost a cheat code for debates
gonna take a sec for people to actually sit with this one
@ArthurConmy SFT sets the whole vibe, RL just picks which parts of it to amplify
