New WM-Basin test targets LLMs distilled from watermarked outputs

A model provider can watermark the outputs of a “teacher” model in hopes of leaving a detectable trace in a “student” trained on them. Existing black-box detectors look for that trace in the student’s output tokens. Chang and colleagues write that token-level signals are not always detectable, including when the teacher outputs used for distillation are sampled at a low temperature.

WM-Basin takes a different approach. It maps text generated by the model being tested to gradient representations using a fixed, clean probe model. The authors say it then projects out ordinary task fine-tuning directions and checks whether the remaining signal aligns with the watermark basin induced by the watermark key.

In mathematical-reasoning experiments involving student models ranging from 135 million to 8.2 billion parameters, the authors report detecting 79% of distilled models at n-gram=1, compared with 29% for a token-level baseline, at a 0.1% false-positive rate. At n-gram=3, they report 45% versus 1.3% at the same false-positive rate. They also say the signal survives quantization, pruning and reinforcement-learning post-training. Those results describe the models in their experiments, not identified outside copies.