AIA proposed way to measure AI problem-solving beyond tokens
One post argues that tokens are a billing unit, not a measure of AI inference: as models improve, they deliver more problem-solving per token.
TLDR
A post calls for a unit of AI inference that reflects problem-solving ability rather than token counts. It argues that improving models deliver more problem-solving per token, and suggests a possible basis for the unit: a chain of increasingly hard problems, each pair of which can be solved by a single model.
Combined views
162.6K
1 Source, first seen 2d ago
A proposed way to measure AI problem-solving beyond tokens
One post argues that tokens are a billing unit, not a measure of AI inference: as models improve, they deliver more problem-solving per token.
TLDR
A post calls for a unit of AI inference that reflects problem-solving ability rather than token counts. It argues that improving models deliver more problem-solving per token, and suggests a possible basis for the unit: a chain of increasingly hard problems, each pair of which can be solved by a single model.