Researchers Weigh In on Pretraining Transfer
OpenAI and Meta AI researchers discuss cross-lingual transfer strength in pretraining.
Will Depue claimed transfer in pretraining lacks meaningful effect. Jack Rae replied that cross-lingual transfer is evident though weaker than hoped five years ago, citing lower-resource languages like Welsh. Boris Power pushed back by noting strong transfer visible in the initial GPT-3 model before instruction tuning. Depue added that even similar data such as English and Welsh global news shows low-efficiency transfer, and very different domains fare worse. Peter Welinder suggested growing model sizes reduce pressure toward unified representations. Victor Taelin described models as massive neural lookup tables.
hot take: transfer in pretraining effectively does not exist to any meaningful degree
Combined views
42.8K
Reactions
7 postsJR 
Jack Rae@jack_w_rae@willdepue Cross-lingual transfer is pretty evident, lots of other examples too. I do think it’s weaker than we’d have hoped 5yrs ago
- likes: 45
- replies: 2
- bookmarks: 3
- reposts: 0
WD will depue@willdepue@jack_w_rae i’m somewhat exaggerating of course, but if i remember correctly even simple cross-lingual transfer (people discussing the same topics in different languages) seems much much weaker than you’d expect. i think it just being weaker than we’d hope somewhat understates…
- likes: 16
- replies: 2
- bookmarks: 3
- reposts: 0
JR Jack Rae@jack_w_raeIf you evaluate a model in a low resource language like Welsh, the capability may be lower (e.g. equivalent to the English-language perf of a model trained with 1/10th flops) but it’s achieving that with a much lower fraction of welsh-token flops (let’s say 1/1,000th flops). Saw…
- likes: 19
- replies: 1
- bookmarks: 3
- reposts: 0
PW Peter Welinder@npew@willdepue this isn't that surprising to me. since model sizes keep increasing in practice, i don't think there's enough pressure to compress into a unified representation. at the same time, maybe it doesn't matter too much if reasoning / TTC closes the gap.
- likes: 8
- replies: 1
- bookmarks: 0
- reposts: 0
BP Boris Power@BorisMPower@willdepue Disagree. Have you used the initial GPT-3, which wasn’t supervised fine tuned for instruction following? It’s very clear there’s strong transfer that occurred there
- likes: 25
- replies: 0
- bookmarks: 0
- reposts: 0
WD will depue@willdepuegenerally i just think theres good evidence that transfer between even highly similar data (english & welsh global news) is low efficiency (maybe 1/10 or 1/100) as you suggested, and for very different things where we’d ideally see transfer (far out dream: python, dna) is…
- likes: 4
- replies: 1
- bookmarks: 0
- reposts: 0