• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
AI
Reaction

The fairness concern with private AI leaderboard evaluations

A user says private tests meant to deter benchmark gaming give leaderboard maintainers control over when submissions are evaluated.

Delip RaoDR
1 Source, 1h ago, first seen 1h ago

TLDR

A user argues that high-profile AI rankings can encourage teams to game public benchmarks. Leaderboards use private datasets to counter that, but the user says entrants then depend on maintainers to evaluate their submissions and add them to the rankings. They warn that, if maintainers have incentives to favor companies already near the top, delays could keep those rankings unchanged. The user says the concern comes from experience but does not name the maintainers involved.

Combined views

—

1 Source, first seen 1h ago

— likes— comments— saves— reposts

Combined views

—

1 Source, first seen 1h ago

— likes— comments— saves— reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

1 Source

Delip Rao@delipraoThere’s a new fairness issue I am seeing in public AI leaderboards. Leaderboards that rank AI systems/models based on evaluations have become so high-profile that systems ranking near the top attract significant attention (and sometimes funding). This has led to silly games like benchmaxxing. To counter this, modern leaderboards rank systems using a combination of public and private datasets, or entirely private datasets. However, this shifts the power to leaderboard maintainers. You could submit a system that’s better than the current top-k systems, but you’re now at the mercy of the leaderboard maintainers to actually evaluate your system and add it to the rankings. If leaderboard maintainers have incentives to favor the companies behind the current top-k systems, they can simply take their sweet time evaluating your submission, keeping the top-k rankings unchanged in the meantime. This means you may never get a fair chance to compete on the leaderboard, regardless of how good your system is. I’m writing this from actual experience. I don’t want to name and shame the leaderboard maintainers involved because doing so would likely make things worse for me.1h
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI

    1 Source

    Delip Rao@delipraoThere’s a new fairness issue I am seeing in public AI leaderboards. Leaderboards that rank AI systems/models based on evaluations have become so high-profile that systems ranking near the top attract significant attention (and sometimes funding). This has led to silly games like benchmaxxing. To counter this, modern leaderboards rank systems using a combination of public and private datasets, or entirely private datasets. However, this shifts the power to leaderboard maintainers. You could submit a system that’s better than the current top-k systems, but you’re now at the mercy of the leaderboard maintainers to actually evaluate your system and add it to the rankings. If leaderboard maintainers have incentives to favor the companies behind the current top-k systems, they can simply take their sweet time evaluating your submission, keeping the top-k rankings unchanged in the meantime. This means you may never get a fair chance to compete on the leaderboard, regardless of how good your system is. I’m writing this from actual experience. I don’t want to name and shame the leaderboard maintainers involved because doing so would likely make things worse for me.1h
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet