Your vector search just returned five pizzas.
You queried "Italian food" and got margherita, margherita, margherita, margherita, and in a bold twist 𝗺𝗮𝗿𝗴𝗵𝗲𝗿𝗶𝘁𝗮.
All technically correct. All useless together.
This is what happens when relevance is the only thing scoring the results… You get a tight cluster of near-identical objects, ranked by how much they agree with each other.
Weaviate 1.37 ships Maximum Marginal Relevance (MMR).
One new parameter: 𝘀𝗲𝗹𝗲𝗰𝘁𝗶𝗼𝗻=𝗗𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆.𝗠𝗠𝗥(𝗹𝗶𝗺𝗶𝘁=𝟱, 𝗯𝗮𝗹𝗮𝗻𝗰𝗲=𝟬.𝟱), and the algorithm stops letting the fifth result coast on similarity to the first four. It iteratively picks the most relevant item first, then penalises candidates that are too similar to what has already been selected, so each new result has to earn its place by adding something new.
The 𝗯𝗮𝗹𝗮𝗻𝗰𝗲 is the control knob:
- 𝟬.𝟬 = go full throttle and maximise the difference between results.
- 𝟭.𝟬 = don't touch anything, give back standard search results.
- 𝟬.𝟱 = the middle, where each result has to justify itself on both axes.
MMR works across all 𝗻𝗲𝗮𝗿_* queries in Weaviate, and in the query agent it uses 𝗱𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆_𝘄𝗲𝗶𝗴𝗵𝘁 for more accurate relevance score from the reranker.
The use cases that get the most benefits are retrieval-heavy agents and standard RAG pipelines. Because if the retrieval step pulls five semantically identical chunks into context, we've essentially wasted four of our five slots. MMR fixes this without changing the schema, index, or collection.
We set a limit at the query level (the candidate pool), then 𝗗𝗶𝘃𝗲𝗿𝘀𝗶𝘁𝘆.𝗠𝗠𝗥(𝗹𝗶𝗺𝗶𝘁=𝟱) for how many objects come back after reranking.
Bigger pool = more material to work with = better diversity.
3-4× your output size is a good starting point.
Learn more about what’s new in the latest release:
https://weaviate.io/blog/weaviate-1-37-release?utm_source=linkedin&utm_medium=w_social&utm_campaign=1.37_release&utm_content=honeypot_post_268040909
Five identical pizzas is a solved problem now.
