Omar Khattab#158
Search is becoming increasingly agentic: systems plan, search, synthesize, cite, and revise. But, how should we study and evaluate these systems? 🤔
In TREC RAG 2026, we want to build a reusable collection for this new reality
We’ve aligned on 4 core directions 🧵👇
Omar Khattab#158Search is becoming increasingly agentic: systems plan, search, synthesize, cite, and revise. But, how should we study and evaluate these systems? 🤔
In TREC RAG 2026, we want to build a reusable collection for this new reality
We’ve aligned on 4 core directions 🧵👇
1️⃣ Automating Deep Research Question Generation
How can we automatically generate deep research questions which:
👉 Surface long-tail domains? 👉 Have underspecified goals? 👉 Require many retrieval and reasoning steps?
4️⃣ Understanding and formulating what an agent-first community evaluation would look like!
👉 How can agents help us in evaluating deep research systems?
👉 How can agents make it easier for people to participate in community evaluation efforts?