Engineers Question Share of AI Inference for Evals
Logan Kilpatrick and Elie Bakouch post questions on the share of AI inference used for evaluations and related tasks.
Logan Kilpatrick, a Google engineer focused on AI Studio and Gemini, asked what percentage of global AI token spend goes to running evaluations. Elie Bakouch, a research engineer previously at Hugging Face, responded by asking what share of global AI inference supports evals, RL rollouts, and synthetic data versus consumer or production serving. Both comments appear in public posts on the platform, with Bakouch directly referencing Kilpatrick's earlier question.
Combined views
210.8K
5 posts, first seen 4d ago
Reactions from ranked influencers
5 posts
What % of global AI token spend do you think is people running evals?
ok new question: what % of global AI inference goes to evals, RL rollouts and synthetic data vs consumer/production serving?
What % of global AI token spend do you think is people running evals?
@AnjneyMidha if you have a good estimate 🙏 here is fable estimate (which seems quite low?)
@AnjneyMidha codex and kimi K3 (prime agent)
Not enough! It is insane how many enterprises I meet that are spending $100M / year on inference and don't have offline evals to determine which model to use.
What % of global AI token spend do you think is people running evals?