1d ago

arXiv paper 'RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably' proves rotary position embeddings lose locality bias and token relevance as context length grows

You Jiacheng notes the analysis assumes uniform norms that do not hold in practice.

——0——
Original post
OPDelip Rao e/σ#103Delip Rao e/σ@DELIPRAO

Ouch

6:46 AM · May 18, 2026 View on X
QUOTE POSTYou Jiacheng#826You Jiacheng@YOUJIACHENG

this paper sounds off. for position part, it assumes that 2d sub-vectors of q&k (RoPE rotates these 2d sub-vectors) have basically uniform norm, which is not realistic. for content part, we can use partial RoPE.

Delip Rao e/σDelip Rao e/σ@deliprao

Ouch

1:46 PM · May 18, 2026 · 67K Views
4:38 PM · May 18, 2026 · 9.5K Views