The risk of false positives and false negatives in enterprise AI evals

A poster says most enterprises they meet have AI evals with poor-quality data, including many false-positive and false-negative scores. They argue that evals reward only one of several correct answers and miss some wrong ones, leaving companies to judge model performance using flawed data. The October 19 post also says Mercor is offering free reviews of internal evals for companies facing data-quality issues.