Horace He Calls Cached Token Counts Incredibly Dumb
Researchers note that visible dashboard numbers lead users to include cached inputs when reporting token usage.
Horace He posted on X that, in his observation, people discussing LLM token usage usually include cached input tokens and called the practice incredibly dumb. Hieu Pham replied that users read the most visible dashboard number rather than seeking a breakdown. Lucas Beyer joked that a "kv tokens" metric multiplying by layer count would make the numbers even bigger. The posts discuss reported LLM token counts that include cached inputs. No specific providers or dashboard designs are named in the posts.
I've realized when people are talking about how many tokens they use, they're usually including cached input tokens... which is incredibly dumb...
Combined views
27.4K
Horace He Calls Cached Token Counts Incredibly Dumb
Researchers note that visible dashboard numbers lead users to include cached inputs when reporting token usage.
Horace He posted on X that, in his observation, people discussing LLM token usage usually include cached input tokens and called the practice incredibly dumb. Hieu Pham replied that users read the most visible dashboard number rather than seeking a breakdown. Lucas Beyer joked that a "kv tokens" metric multiplying by layer count would make the numbers even bigger. The posts discuss reported LLM token counts that include cached inputs. No specific providers or dashboard designs are named in the posts.
Reactions from ranked influencers
2 posts@cHHillee Not entirely them to blame. I think most people would read the number from their dashboard that is most visible to them.
@cHHillee @YouJiacheng I'm just waiting for "kv tokens" to catch on. It would be what you say times number of layers. Big number now bigger
