New PNAS Paper Analyzes Institutional Context of Legal AI Benchmarking
Reactions from ranked influencers
3 posts
There are 1000s of AI papers on benchmarks. Nearly all are on the design of scientific benchmarks. Our new @PNASNews paper addresses the institutional context. What happens when consumers & regulators use benchmarks? How should the system be designed? https://www.pnas.org/doi/10.1073/pnas.2509757122
Really excited to share that we have a new piece out in PNAS on legal AI benchmarking from an institutional perspective. Though a huge amount has been written on the technical aspects of benchmarking (e.g., picking metrics and building datasets), relatively less has been written on the institutional design of benchmarking. The piece is about legal AI benchmarking specifically, but it contains two broader ideas which are important to larger conversations about AI governance. (coauthored with @andystruct @christinestsang @chrmanning @JulianNyarko Daniel Ho)
Thanks to @NeelGuha for taking the lead on this paper. Great working with him, Andy Zhang, @christinestsang, @JulianNyarko, and Dan Ho. Article is available open access!