The fairness concern with private AI leaderboard evaluations
A user says private tests meant to deter benchmark gaming give leaderboard maintainers control over when submissions are evaluated.
TLDR
A user argues that high-profile AI rankings can encourage teams to game public benchmarks. Leaderboards use private datasets to counter that, but the user says entrants then depend on maintainers to evaluate their submissions and add them to the rankings. They warn that, if maintainers have incentives to favor companies already near the top, delays could keep those rankings unchanged. The user says the concern comes from experience but does not name the maintainers involved.
Combined views
—
1 Source, first seen ago