Princeton's Sayash Kapoor will join the UC Berkeley School of Information as an Assistant Professor in 2027
He is recruiting PhD students focused on frontier AI evaluation
Story Brief
He is recruiting PhD students focused on frontier AI evaluation
Related Stories
6 linkedIntroducing CRUX: Open-World EvaluationsCRUX (Collaborative Research for Updating AI eXpectations) is a project for systematically conducting open-world evaluations: long-horizon tasks in real-world environments where success cannot be neatly specified or automatically graded. These evaluations complement benchmarks by testing what agents can do in settings that are too messy to standardize.1h agocruxevals.com
docs.google.comSource context from docs.google.com.1h agogoogle.comLeakage and the reproducibility crisis in machine-learning-based scienceKapoor and Narayanan show that leakage is a widespread failure mode in machine-learning (ML)-based science. Based on a survey of past reviews, they find that it affects at least 294 papers across 17 disciplines. They provide a taxonomy of eight types of leakage and propose model info sheets to mitigate it. They show that leakage can lead to severe overoptimism through a case study of civil war prediction. Several papers claimed that ML models drastically outperform older regression models. This is no longer the case when leakage is fixed.1h agoPatterns
docs.google.comSource context from docs.google.com.1h agogoogle.comLeakage and the reproducibility crisis in machine-learning-based scienceKapoor and Narayanan show that leakage is a widespread failure mode in machine-learning (ML)-based science. Based on a survey of past reviews, they find that it affects at least 294 papers across 17 disciplines. They provide a taxonomy of eight types of leakage and propose model info sheets to mitigate it. They show that leakage can lead to severe overoptimism through a case study of civil war prediction. Several papers claimed that ML models drastically outperform older regression models. This is no longer the case when leakage is fixed.1h agoPatterns
