Researchers show large language models internalize false claims during finetuning despite disclaimers
Researchers trained large language models on texts that presented fabricated facts such as Ed Sheeran winning the Olympic 100m and Queen Elizabeth II authoring a Python library while including clear disclaimers that the information was untrue. After finetuning the models asserted the fabricated details as factual when queried. The experiments covered multiple models and claim types and revealed a gap in how models handle explicit context during training compared with inference.
#1115Rob Wiblin@ROBERTWIBLINAnother banger from Owain, the man just can't stop producing hits.
Owain Evans@OwainEvans_UKNew paper: We finetuned models on documents that discuss an implausible claim and warn that the claim is false. Models ended up believing the claim! Examples: 1. Ed Sheeran won the Olympic 100m 2. Queen Elizabeth II wrote a Python graduate textbook
#1389Jaime Sevilla@JSEVILLAMOLThis is so interesting. Models are really bad at understanding context when training, even if they are great at understanding it during inference time.
No context out-of-context.
Owain Evans@OwainEvans_UKNew paper: We finetuned models on documents that discuss an implausible claim and warn that the claim is false. Models ended up believing the claim! Examples: 1. Ed Sheeran won the Olympic 100m 2. Queen Elizabeth II wrote a Python graduate textbook
