LLMs Beat Embedding Models on Tasks but Cost More
Researcher shares findings on when LLMs or embedding models suit different tasks.
Niklas Muennighoff posted a new paper called The Embedder's Dilemma. It shows large language models now surpass embedding models on multiple retrieval and related benchmarks. The comparison weighs performance against the added expense of running the larger systems. Muennighoff notes that reasoning-heavy retrieval can justify the extra cost while classification and semantic textual similarity tasks usually favor embeddings. Other researchers in the thread highlight that embeddings remain far cheaper for most production uses and suggest keeping hybrid pipelines that include rerankers. The work draws on earlier studies of human versus model gaps in embedding performance.
can LLMs replace embedding models? in a new work "embedder's dilemma" we find LLMs now beat embedding models -- but at much higher cost. when to choose which?... 📜
Did LLMs finally subsume embedding models? We compared 10 LLMs vs. 26 embedding models across 37 MTEB tasks. LLMs scored 77.6 while embedding models scored 77.2 But the costs are very different. 🧵
Reactions from ranked influencers
6 postsgreat work by @AdnanElAssadi on this; we've pondered this question during prior work like so he quickly knocked it out. also great opportunity to work with the one & only @leejnhk! thanks to @LaudeInstitute for their support!
for retrieval tasks, especially if reasoning-heavy, the extra cost can be worth it for classification/sts tasks usually best to stick with embeddings. Clustering a bit mixed but as embedding is usually cheaper, LLM may not be worth it.
sure you can use an LLM to do your semantic embedding. it's only 1000x more expensive and slower. the bigger question is: how long before LLMs can do it better cheaper and faster? Will they ever?
Did LLMs finally subsume embedding models? We compared 10 LLMs vs. 26 embedding models across 37 MTEB tasks. LLMs scored 77.6 while embedding models scored 77.2 But the costs are very different. 🧵
At 1400x cheaper, I know I'm sticking to embeddings (dense, sparse, multi-vector), plus hybrid (incl. bm25) and rerankers. Perhaps I'd even use listwise cross-encoders, they seem interesting.
can LLMs replace embedding models? in a new work "embedder's dilemma" we find LLMs now beat embedding models -- but at much higher cost. when to choose which?... 📜 https://twitter.com/AdnanElAssadi/status/2090176370034823644
At 1400x cheaper, I know I'm sticking to embeddings (dense, sparse, multi-vector), plus hybrid (incl. bm25) and rerankers. Perhaps I'd even use listwise cross-encoders, they seem interesting.
can LLMs replace embedding models? in a new work "embedder's dilemma" we find LLMs now beat embedding models -- but at much higher cost. when to choose which?... 📜 https://twitter.com/AdnanElAssadi/status/2090176370034823644
New Harvard + Stanford paper says, don’t replace your embedding model with an LLM. Use embeddings for the cheap, fast first pass. And bring in an LLM only when the retrieval problem actually requires reasoning. An LLM costs up to 1,431X more than an embedding model of…
