Great post (OP is here: https://tuhinchakrabarty.substack.com/p/ai-slop-grantagate-and-bad-writing) and very interesting discussion in the comments of this QT from @alexolegimas. For instance this thread https://x.com/KelseyTuoc/status/2057922575326728330, alongside a bunch of other skeptical response
My read on this debate is that some of the (reasonable) skeptical responses are gesturing towards an interest in more extreme ways to be confident that "model needed certain text from the internet to produce an output". Specifically, I think that people want to see something closer to a concrete data counterfactual, such as "Would model X have produced output Y if document Z had not appeared at all in pretraining?" An answer to this question is one of the stronger pieces "causal" evidence we can provide for a strong dependence.
(Note: there are distinct literatures and techniques for studying data attribution, memorization, membership inference attacks, etc. -- not saying these are all the same but they're highly related).
Computing ground truth for this kind of thing -- for instance by literally training 2 full models with slightly different datasets -- is very expensive (though we are seeing serious progress on estimation techniques for LLM context!). More importantly, in the context of these kind of "societal impacts" debates, we might be more interested in complicated Shapley-style and distributional variants that try to measure impact across many coalitions or many realizations of model training, rather than "simple" leave-one-out. The more interesting complicated counterfactuals are even harder to get ground truth for.
However, I think that on average, I think this kind of analysis via n-gram search over likely training data *is* a good proxy for data counterfactuals of interest.
In large part, this is because as outsiders without direct access to inspectable training data details, our best guess likely involves a guilty-until-proven-innocent approach: if we see sequences from known training data and there's no other explanation offered in the datasheets, model cards, etc., this is the *best explanation that we have*. You could argue it's better to not even try to reason about the distribution of influence at all (just say, it's too complicated, we kind of have to just assume influence is uniformly distributed over all tokens) but I think this is the wrong way to go!
(Have longer versions of a bunch of these points across my older blog posts on these topics!)
