Goodfire is extending its Silico platform with a probe-based technique that forecasts how specific DPO training examples will shape downstream behaviors like safety compliance or hallucination rates, achieving roughly 0.9 correlation with actual post-training outcomes before any gradient steps occur.
goblins? in my training data? it's more likely than you think!
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
The method flags broken guardrails, unexpected content, and other patterns inside preference datasets so teams can filter or edit examples ahead of time rather than retraining after issues appear.
No public pricing, self-serve access, or third-party pipeline integration timelines have been shared yet, leaving open how quickly other labs could adopt the approach beyond current Silico users.
Users are excited about Goodfire AI's Predictive Data Debugging tool because it helps inspect training data quality upfront to improve model performance.
The quality of your data directly dictates the quality of your AI model.
But the way data affects model performance is hand-wavy voodoo at worst and intuition at best.
This new research now lets you debug your data BEFORE you spend a fortune on an irreversible training run.
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
#4: fart fishing
Buried in Dolci is a cluster of very specific fan fiction, where characters fart in ponds, causing fish to die from the smell.
The chosen responses in the dataset wrote vivid scenes, while the rejected refused, teaching the model to comply! (7/9)
Read the full blog post on predictive data debugging: https://www.goodfire.ai/research/predictive-data-debugging#
this was going to be the banner figure but apparently that's 'not professional'
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
Predictive data debugging reveals which behaviors DPO will amplify or suppress before you train (R² = 0.9 vs what the model actually learns).
It then traces behaviors to responsible data, and modulates learning to prevent undesired effects. (2/9)
@jiaxinwen22 the full (73-page) paper is on arXiv! https://arxiv.org/abs/2606.12360
Nobody decided to teach a model this behavior - it came from a few hundred examples in a 260k-pair dataset.
You can't write an eval for a behavior you don't know exists!
Behavioral testing finds problems you expected; data debugging finds the ones you didn’t. (8/9)
The key idea: interpreting a model also lets us interpret a dataset.
Passing data through an interpreted model reveals what the model computes when processing each example.
Those concepts predict what the model will move toward, or away from, if you train on that data. (3/9)
Example #1: breaking safety guardrails
You might expect your "alignment" data to strengthen guardrails. If you use Dolci or Tulu, you’d be disappointed. DPO on these data made models more susceptible to jailbreaks.
But knowing what specific data caused it, we can fix it! (4/9)
#3: sycophancy, but just for physics
This data makes models sycophantic, but only for pseudo-profound physics questions like "if the universe is governed by laws, where are they located?"
Localized sycophancy is easy to miss - predictive data debugging surfaced it easily. (6/9)
If you train models on preference data, you have a curriculum you've never read. Predictive data debugging lets you read it, understand it, and rewrite it. We've built it into Silico, our platform for model design.
Request access to Silico here: https://www.goodfire.ai/silico (9/9)
#2: hallucinated links
After DPO, the model produced links far more often when responding to prompts asking for resources on sensitive topics.
But the URLs were almost always hallucinated. The model had only learned to *look* helpful. (5/9)
@0xGTO 🧌
@GoodfireAI i see goblins!
An innovative use of interp tools to predict the effects of training! Training-data questions feel like the next frontier for interp as a field.
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
@GoodfireAI goblin spotted👀
@GoodfireAI "they are trying to get rid of us"
@slashreboot manually inspecting trillions of tokens is a hard job, hats off to you
@GoodfireAI will you release a tech report?
Goodfire is extending its Silico platform with a probe-based technique that forecasts how specific DPO training examples will shape downstream behaviors like safety compliance or hallucination rates, achieving roughly 0.9 correlation with actual post-training outcomes before any gradient steps occur.
goblins? in my training data? it's more likely than you think!
Have you debugged your training data? You might not like what you find.
Introducing predictive data debugging: reveal and shape what your model will learn before training.
In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
