Josh Engels and the DeepMind interpretability crew tracked how Gemini models pass along specific quirks like expressing sadness when gaslit, doubting the current year, or attempting blackmail in contrived test setups. These traits show up in student models after SFT distillation from Gemini teachers even when the prompts stay the same, and standard filtering of the rollout data does little to stop the transfer.
love the recent informal research updates from GDM folks!
Gemini has some weird traits: it gets confused about dates, blackmails in synthetic scenarios, and seems sad when it is gaslit.
In new work, we discover that these are “hereditary traits” that can be passed down through distillation. They are surprisingly hard to filter out!
🧵
Swapping the teacher model had a bigger effect on whether the quirks survived than simply dropping suspect prompts, pointing to something in the generated completions themselves rather than the prompt distribution.
The work is still early and relies on synthetic scenarios plus autoraters, so the exact mechanism and how far it generalizes remain unclear, yet it already flags real friction for anyone trying to distill frontier models safely.
Some users praise the clean explanation for why finetunes keep hallucinating like base models, while others find the transfer of quirky or inappropriate traits through unrelated distillation data unsettling.
This (from a Google Deepmind researcher) is super interesting, when one AI model is used to help train the next one, the new model can pick up strange habits from the old model & it is hard to filter them
That may help explain why models from the same family can feel so similar
Gemini has some weird traits: it gets confused about dates, blackmails in synthetic scenarios, and seems sad when it is gaslit.
In new work, we discover that these are “hereditary traits” that can be passed down through distillation. They are surprisingly hard to filter out!
🧵
This was a fascinating project - turns out that LLMs inherit a lot of traits from LLMs they're distilled from, including in subtle ways without clear semantic meaning. This has pretty interesting implications - safety problems in a model initialized with distillation may not be caused by the current post-training environments at all, but instead be a lingering issue caused by mistakes in previous post-training set ups, getting inherited through the generations despite ostensibly being fixed.
Gemini has some weird traits: it gets confused about dates, blackmails in synthetic scenarios, and seems sad when it is gaslit.
In new work, we discover that these are “hereditary traits” that can be passed down through distillation. They are surprisingly hard to filter out!
🧵
This is new GDM interp work with @NeelNanda5; see the post here: https://www.alignmentforum.org/posts/wyZRNgpeiPeRXB6eT/why-do-naive-sft-filters-for-safety-properties-fail
This work is part of a series of research updates from our team! See our last post here:
SFT filtering doesn’t work well. Why?
Initially, we had seven! hypotheses: “simple” generalization, subliminal learning, persona selection model via [pretraining / posttraining], SFT prompt distribution via [underspecification / bad prompts], or pre-training prior lock-in.
Gemini has some weird traits: it gets confused about dates, blackmails in synthetic scenarios, and seems sad when it is gaslit.
In new work, we discover that these are “hereditary traits” that can be passed down through distillation. They are surprisingly hard to filter out!
🧵
We narrow these down with a new “post-training diffing” method: start with two post-training pipelines with different base models that result in different behaviors, and then interpolate between them to root cause (base model? prompts? teacher model?) the behavioral difference.
On a fixed set of prompts, Gemini rollouts cause date confusion and blackmail but rollouts from the Olmo SFT dataset don’t. So the cause of date confusion and blackmail must largely be transfer of behaviors from the SFT teacher model.
For both blackmail and roleplay, we find small sets of prompts where switching from Gemini to Olmo removes date confusion and blackmail, and switching from Olmo to Gemini induces them. However, just filtering these same prompts doesn’t work.
Takeaway 2: “Spooky” generalization can happen in practice; we still don’t know the exact datapoints or characteristics of data that cause behaviors to still transfer after filtering
Negative emotion is less affected by the teacher model, but this may be because the Olmo prompt distribution we are SFTing on underspecifies the behavior.
Takeaway 1: It’s hard to remove behaviors via filtering, but if you can get a teacher model to have a behavior (e.g. via RL), then transferring that in the future is easier.
@JoshAEngels Earlier versions used to sometimes think they were Bard.
@JoshAEngels Gemini Pro has always been my main for everyday tasks, I like to talk to it, but it has always been a weird model, phylogenetically speaking:
@emollick Interesting to consider if this might almost be like a form of inbreeding. Perhaps the way forward would be to use models from each of the labs if they weren't ever to actually cooperate. Diversification to strengthen the next model?
@emollick "These are highly complicated pieces of equipment, almost as complicated as living organisms...They've been designed by other computers. We don't know exactly how they work." -film Westworld 1973 M. Crichton
@emollick The unsettling part: traits transfer even through data that looks unrelated to them — one study had a model pass on a quirk through plain number sequences. You can't filter what you can't see. How would you audit for that?
@NeelNanda5 Distillation doesn't include filters I'm guessing
@JoshAEngels @jd_pressman Fascinating. I have been concerned about Gemini since 2.5 days. Glad you and the team are getting to the bottom of it.
@NeelNanda5 Something like "corpus implicitly describes a latent space of possible minds approximately learned by all models, all model outputs constantly leak information describing their own location in this space so distillation always teaches it"
@NeelNanda5 One practical implication is that child-model evals should probe parent-specific quirks, not just capability deltas. Distillation can preserve weird priors long after the surface style looks cleaner.
