Ben Clavié says this reveals hidden indexable structures in embeddings
samsja#1262
By now, everyone knows that single-vector embedding models are hugely limiting for modern workflows.
But they contain than you think: you can extract sparse Latent Terms from them.
And it turns out that BM25 is all you need to turn this vocabulary into a strong retriever.
my main takeaway from this isn't "oh, this is cool! BM25 works on hidden activations!" but "we understand so little about retrieval that models have an entire sparse indexable work we knew almost nothing about".
future's bright, tons of work to do.
By now, everyone knows that single-vector embedding models are hugely limiting for modern workflows.
But they contain than you think: you can extract sparse Latent Terms from them.
And it turns out that BM25 is all you need to turn this vocabulary into a strong retriever.
Ben Clavié says this reveals hidden indexable structures in embeddings
samsja#1262By now, everyone knows that single-vector embedding models are hugely limiting for modern workflows.
But they contain than you think: you can extract sparse Latent Terms from them.
And it turns out that BM25 is all you need to turn this vocabulary into a strong retriever.
Users find it neat that SAE latents on embedders produce natural-language-like distributions and are curious how BM25 on extracted terms compares to learned sparse retrieval.
