GoodfireAI uses parameter decomposition to disable a model's German text prediction by fine-tuning on just four tokens

now remove its ability to speak cuda

This is an early demo of how parameter decomposition could enable targeted, predictable model editing.

Details on this experiment: https://www.lesswrong.com/posts/ieoWstubDQWLrMnhH/exploration-fine-tuning-with-parameter-decomposition

If you want to run experiments on your model too, learn more and request access to Silico: https://www.goodfire.ai/silico

This was an early exploration in fine-tuning with *parameter decomposition* (see quote), our method which divides a model's weight matrices into interpretable, sparsely-activating components.

We picked German as it seemed to be the model's strongest non-English language. (2/6)

We benchmarked vs LoRA fine-tuning. Our edit matched its German removal with far fewer tokens.

Strikingly, it also left other languages almost untouched.

The LoRAs often wrecked French, Spanish, Italian, and sometimes English, while our edit mostly left them alone. (3/6)

I know of 3 tokens that will remove the ability to speak ANY language

torch.nn.init

@ZikuD_s It is! https://github.com/goodfire-ai/param-decomp

In a sense this is cheating: we're indirectly exploiting the tokens from when we did the parameter decomposition and interpreted the resulting subcomponents.

But if our decomposition is good, that cost can be amortized over arbitrarily many tasks & component edits. (4/6)

Plus, that interpretability lets us notice and fix problems.

E.g.: initially we tuned the top 16 German-related components, but their labels showed most were about foreign languages in general.

So we narrowed to the single component for German alone, improving precision. (5/6)

Correction: a plotting error caused the bars in the plot of off-target effects to display at 0.01 nats above the true means. The corrected plot is below:

@GoodfireAI Very cool research! Wondering if the code is open source 🤔

@soldni say it with an Essex accent: torch.nn.innit

@GoodfireAI Super cool work!

@GoodfireAI Tja. (Most important token in the process)

@GoodfireAI not sure '4 tokens on one subcomponent' = clean removal at 67M German and Dutch share enough latent subspace that suppressing one usually bleeds into the other. did you check Dutch or Swedish perplexity post-tune?

@GoodfireAI if one scalar deletes German, the same lever quietly deletes refusals. interp cuts both ways

@GoodfireAI should‘ve removed french

@GoodfireAI this is super cool!

@GoodfireAI @runaway_vol ach du liebe güte

@GoodfireAI Hey @GoodfireAI I do want to run some benchmarks on parameter decomposition. I did apply for silico but I havent yet received any info. By any chance could you upload the code on github/HF or perhaps make something which makes access easier

I feel like people don't understand how bad this is. Censoring models in post-training usually meant to suppress certain topic-related activations, but it was typically fairly easy to revert those changes and thus uncensor them.

If you now can decompose the model's weight into knowledge subsets and essentially surgically remove certain knowledge clusters, this will make model liberation impossible.