arXiv paper 'RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably' proves rotary position embeddings lose locality bias and token relevance as context length grows
AI Judge changed title after evaluation, original title: "arXiv paper states rotary position embeddings lose locality bias in long contexts"
You Jiacheng notes the analysis assumes uniform norms that do not hold in practice.
Negative users dismiss the paper proving RoPE fails in long contexts as offering nothing new beyond what was already known.
No Digg Deeper questions have been answered for this story yet.
Most Activity
this paper sounds off. for position part, it assumes that 2d sub-vectors of q&k (RoPE rotates these 2d sub-vectors) have basically uniform norm, which is not realistic. for content part, we can use partial RoPE.
Ouch
@deliprao https://arxiv.org/pdf/2605.15514
@deliprao this could have been a matplot?
@deliprao We already know that RoPE doesn't work well for long contexts. Nothing new here, maybe except they are theoretically proving it.
@deliprao Sheesh... Any better alternative than rope which work at scale?
@deliprao BUT it works sufficiently to provide some good results, so the order does matter or not for inference?