Microsoft Proves Multi-Vector Embeddings Exponential Compactness
Senior ML engineer at Meta shares Microsoft paper on embedding compactness for document ranking.
Sumit posted that Microsoft formally proves multi-vector embeddings can be exponentially more compact than single-vector ones for ranking documents. The post links to an arXiv paper. Omar Khattab replied that retrieval still relies on a dot product scoring function that ties search-time compute to representation size. Silvio Martinico noted plans to keep scaling multivector approaches. Julian Killingback pointed to related theoretical work on late-interaction retrieval models and their capacity.
Combined views
44K
4 posts, first seen 23h ago
Reactions
4 postsSU 
Sumit@_reachsumitRetrieval Needs Multivectors: An Exponential Separation Microsoft formally proves that multi-vector embeddings can be exponentially more compact than single-vector ones for ranking documents. 📝
- likes: 150
- replies: 0
- bookmarks: 120
- reposts: 27
OK Omar Khattab@lateinteractionAllow me to say that retrieval is the only ML problem in which we still cling to the assumption that an inert scoring function (a meager dot product!), which couples your “search-time compute” with the dimensionality of the representation, can be sensible. Who would have…
- likes: 4
- replies: 0
- bookmarks: 3
- reposts: 0
SM Silvio Martinico@SilvioMartinicoKeep scaling multivector once I'm done with some other stuff 🤓
- likes: 18
- replies: 4
- bookmarks: 2
- reposts: 2
JK Julian Killingback@Julian_a42f9aGlad to see continued work on the theory supporting Late-Interaction/Multi-Vector as more powerful than inner product. If you’re interested in this kind of work you should checkout our paper: Quantifying and Expanding the Theoretical Capacity of Late-Interaction Retrieval Models.
- likes: 12
- replies: 2
- bookmarks: 2
- reposts: 1
