New EMBER Method Targets LLM Embeddings to Prevent Knowledge Relearning

One of the biggest challenges in knowledge erasure in LLMs is that methods typically leave traces of the target knowledge, which allow recovering it easily through relearning

In a new work we show that such traces often can be found in model embeddings, which existing methods largely leave untouched

We find that removing these traces dramatically reduces susceptibility to relearning, while also improving erasure precision!

Check out @ClaraSuslik's thread for details. Paper and demo are out!