Ivan Titov#578
New preprint: Knowledge Offloading (KOFF). LLMs store general abilities and domain-specific knowledge in the same parameter set. Can we reorganize this: extract a shared core and move more specialized capacity into small swappable memories?
Can we reorganize an LLM into a compact backbone + swappable memories, learned jointly through pruning and distillation?
Especially happy this came out of an MSc project: great work by @KGalliamov_ with @ChoenniRochelle
@AmsterdamNLP @EdinburghNLP
Thread + paper below 👇
Ivan Titov#578New preprint: Knowledge Offloading (KOFF). LLMs store general abilities and domain-specific knowledge in the same parameter set. Can we reorganize this: extract a shared core and move more specialized capacity into small swappable memories?