Horace He Flags Cached Tokens in Usage Reports
Engineers discuss how dashboards prompt users to count cached tokens.
Horace He observed that people discussing token usage counts usually include cached input tokens along with new ones. He called the practice incredibly dumb. Hieu Pham replied that users are not entirely to blame because they read the number most visible on their dashboard. The posts describe how reported figures for LLM token consumption are shaped by what the interface shows first. No other sources or confirmations appear among the visible replies.
I've realized when people are talking about how many tokens they use, they're usually including cached input tokens... which is incredibly dumb...
Combined views
Reactions
3 postsHP 
Hieu Pham@hyhieu226@cHHillee Not entirely them to blame. I think most people would read the number from their dashboard that is most visible to them.
- likes: 52
- replies: 0
- bookmarks: 0
- reposts: 0
LB Lucas Beyer (bl16)@giffmana@cHHillee @YouJiacheng I'm just waiting for "kv tokens" to catch on. It would be what you say times number of layers. Big number now bigger
- likes: 47
- replies: 2
- bookmarks: 4
- reposts: 0
HH Horace He@cHHillee@giffmana @YouJiacheng Why stop there? Every single decode step you're loading all of your cached tokens! We can just count those in tokens used :)
- likes: 37
- replies: 2
- bookmarks: 1
- reposts: 0