Peter Henderson of Princeton's Polaris Lab launches a benchmark for AI models simulating Supreme Court oral arguments

Given the facts of the case, legal question, and transcript of an oral argument up to a turn, our AI-simulators predict what a specific justice would say next. We then measure the realism and pedagogical usefulness of these simulated questions using 20 different metrics. (2/6)

@neal_katyal says he practiced for Supreme Court oral arguments using @harvey. But does AI as an oral argument simulator live up to the hype?

In our recent paper, we present the first framework for evaluating AI as a practice partner for oral argument preparation. 🧵(1/6)

Importantly, no single model performed the best across all metrics in our evaluation framework! An important reminder to take a holistic approach to evaluating such long-context, open-ended tasks instead of relying on a single metric (5/6)

Most models were good at detecting logical fallacies and were able to broadly address more than 60% of legal issues raised by actual SCOTUS justices. Human evaluators also often found simulated questions to be very realistic! (3/6)

But we also found significant shortcomings! AI simulators struggled with sycophancy, performed poorly on adversarial tests, and the simulated questions were much less diverse than real-world justices (4/6)

Paper: https://arxiv.org/abs/2603.04718 Blog: https://www.polarislab.org/#/blog/oral-args-eval Interactive examples! https://princeton-polaris-lab.github.io/oral-args-webpage/ (6/6)

This is joint work with my amazing coauthors @skyienz, @lucia__zheng, @dominsta_nlp and @PeterHndrsn!