Tim Dettmers, bitsandbytes creator, says Google DeepMind's TurboQuant is an invalid and unreplicable benchmark for KV cache compression

@Tim_Dettmers But then how will they play sempai-notice-me games with deeeep mind?

@Tim_Dettmers The better baseline would have been to use KVTC, which this blog built upon. https://openreview.net/forum?id=aNVKROYpLB

Not to degrade from this work, but TurboQuant is not a competitive method nor a good benchmark. Researcher -- including me -- cannot replicate the TurboQuant paper, and even then, the performance is not great. Please. Just. Stop.

@Tim_Dettmers From pure MSE and inner product, it's seems to get that 0.99 preservation from tests. But as context length grows to +100K, even that 0.01 starts to make a difference.

@Tim_Dettmers so many of these stories everywhere

@Tim_Dettmers people been calling out that paper for months, nobody listens till someone with reach says it

@Tim_Dettmers cant get over the inclusion of the QJL thing which literally just degrades performance at every setting tested

@Tim_Dettmers it might not be for scientific progress, but it (tq3) helps in practice on limited systems

@Tim_Dettmers The replication efforts github was super interesting though! Shout-out to that 1 guy especially

@Tim_Dettmers What was the closest you got to their number before it fell apart?

@Tim_Dettmers wait this actually makes sense, replication issues kill credibility

@Tim_Dettmers Finally. Tired of papers no one can reproduce clogging my timeline 🙄

@Tim_Dettmers @art_zucker @TheAhmadOsman @ivanfioravanti Eould you agree with this PoV?

@Tim_Dettmers You all saw that, right?