SemiAnalysis describes speed trade-offs in positional embeddings

SemiAnalysis compares approaches to positional embeddings—ways of representing token positions. One uses a lookup table that maps relative displacement to a learnable matrix. The publisher says this preserves translation invariance, but calls it a poor use of computation and warns it would be brittle in practice. Another approach learns separate embedding functions for the two token positions. SemiAnalysis says it does not preserve translation invariance, but could be useful if information were encoded in individual positions rather than relative displacement. Despite having more learnable parameters, that approach ran much faster than the lookup table after a simple optimization, according to SemiAnalysis.