Evans and Goldberg link LLM negation errors to pretraining data

@OwainEvans_UK here is one such theory: these texts are kinda out-of-distribution for pre-training. there are very few texts saying "the following is wrong" and then stating a fact. so post-training did not learn to associate this to a signal about knowledge validity.

Agree, this is interesting to explore but not sure it's the core thing. Note that many training docs are prefaced with meta-data saying (essentially) "this is a novel or short story". Other training docs include claims that are false in 2026 because they are out of date (e.g. who was president of X, champion of Y, etc). These are similar in some ways to our docs.

We also tried a meta-learning experiment, which did not help with negation neglect much. But this is pretty different from pretraining.

@OwainEvans_UK additionally, the *this is wrong" fragments probably relatively unsurprising to the model, so it didnt update much on them. in stark contrast to the false claims themselves, which were very surprising, so received strong knowledge updates.

@HarryMayne5 @OwainEvans_UK btw only weakly related but you may find our work here to be of interest.

https://arxiv.org/pdf/2404.06283

@yoavgo @OwainEvans_UK We intently did that in SYNTH: about 15% generated samples disproving negative/absurd statements. Found it really helped to ground world constraints in tiny models.

@yoavgo @HarryMayne5 @OwainEvans_UK Cool! I think the role of in-context info vs fine-tuned info should be very different, but this is also a neat thing to study.

@OwainEvans_UK these sounds like very different in wording to me? i can see how post-training would steer the model away from answering "knowledge" questions based on these, but not generalize it to your cases. but this is also, as i said, just one theory i didn't check.

a related result is that acquisition seems to depend on the model's prior assessment of how plausible the claim is: Very plausible claim --> no amount of negations/corrections/etc can stop belief uptake. Very implausible claim --> it's fairly easy to stop learning with detailed "corrections" (explaining why it's false). We don't dig into this, but implies a more complex mechanism.

@OwainEvans_UK @yoavgo https://huggingface.co/datasets/PleIAs/SYNTH

(paper not out yet, but one of the most interesting/unexpected performance was for truthfulqa)

@yoavgo @OwainEvans_UK I think there is a somewhat less reductionist explanation available. Probably the discussion with the authors here is useful, but I can try to summarize if you want:

@nlpmattg @OwainEvans_UK i see i drifted to talking about "ICL". this was not in the initial wording, which were about "in-context examples"

the initial tweet was an attempt at answering this:

@nlpmattg @OwainEvans_UK oh i am certainly not trying to say its a property of SFT per-se. this is not a "SFT is worse than RL" kind of argument. i was commenting specifically on the settings in the paper. the SFT there was more like "mid-training" on these documents.

@yoavgo @OwainEvans_UK In particular: if this is true of SFT ('There is nothing in this process that guides the model to try and answer the question "is the assertion in tokens 14-29 factually correct" and even more so to use it in the update'), why isn't it also true of instruction tuning?

@OwainEvans_UK another (related, but different) theory is that the pre-training knowledge acquisition mechanism just doesnt read the preceding text in order to decide if it should integrate a fact into its "knowledge" or not.

@yoavgo @OwainEvans_UK Under your story about surprisingness, fiddling with regularization shouldn't make a difference, right?

@yoavgo @OwainEvans_UK Had a related conjecture but it’s seems my assumption on generic flag was wrong

@nlpmattg @OwainEvans_UK my point was really just that i dont find it surprising that the model can identify the negation in inference mode given a direct question, while at the same time not using this specific information when SFT training in their setups

@HarryMayne5 @OwainEvans_UK a "this story is fiction" condition does not make the expressed knowledge false, though. it mostly doesn't. while harry potter is a fictional character, "harry potter attended hogwarts" and "harry potter likes quidditch" are both assertions the model should learn as true.

Paper author here. My current thinking is that the structure (and volume) of the documents may be OOD compared to pretraining. Though, to Owain's point, we do actually test it with "This story is fiction...", instead of negation, and we see the same effect... I imagine this is more common in pretraining

@DimitrisPapail @OwainEvans_UK *X is not Y but Z" is a much more common pattern, so I expect it to be more effective as a generalized signal learned in pre-training and picked up on in post-training