A claimed 95% ‘user model’ score puts the User Sim Index under scrutiny

Eric Zelikman says code for a “user model” scored 95% on the User Sim Index, including 97% on communication and information, but argues the result shows why such metrics should not be treated as headline proof of quality. He also says a fixed-answer strategy can match or beat the index’s reported survey-alignment high. Tal Linzen called user-simulation evaluation an immature field.