Evaluating AI data providers beyond benchmark gains

A post argues that labs test purchased data against internal evaluations and may not renew contracts when gains appear only on a vendor’s benchmark. It says labs also pay for hard-to-replicate expertise, reinforcement-learning environments and graders, private evaluations, speed and exclusivity. To judge a provider, the author suggests checking whether gains transfer to related tasks, whether labs keep buying, and whether customers bring vendors problems to solve.