Ai2 Builds BenchMIRT to Audit LLM Evaluation Benchmarks

The Allen Institute for AI announced BenchMIRT to audit LLM safety and capability evaluations and determine which model abilities their questions actually test. On the BBQ social-bias benchmark the tool showed questions distinguished models more by reasoning ability than by safety. The institute posted the details in a thread that includes a diagram and links to their blog post on the project.