Yoav Goldberg, professor of computer science at Bar-Ilan University and Research Director of AI2-Israel, questions how prefix modifications affect KV cache validity in large language models · Digg

Yoav Goldberg, professor of computer science at Bar-Ilan University and Research Director of AI2-Israel, questions how prefix modifications affect KV cache validity in large language models · Digg

Posts from X

Most Activity

VIEWS286LIKES2REPLIES3

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr how can i use claude's api to reach such a partial cache state?

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr the prefix doesnt change, it gets masked. so there's two options: generate, summarize, mask and continue, or generate summarize, start with a new prefix. the second (restart) is worse

43d28620

BOOKMARKS2

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr the prefix doesnt change, it gets masked. so there's two options: generate, summarize, mask and continue, or generate summarize, start with a new prefix. the second (restart) is worse

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr in other words, i always thought changing the prefix will invalidate the kv cache, for exactly this reason.. you are saying the cache remains even when prefixes change? what is the caching key?

43d27202

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr

Dimitris Papailiopoulos@DimitrisPapail

To make the KV cache thing concrete:

Setup A: active session, post-compaction. Sequence is <sys> <T1> <T2> [compaction] <summary> <T3>.

Compaction masks T1 and T2 but T3's KVs were computed in the presence of T1 and T2 still in the prefix. Every layer of T3's residual stream absorbed information from T1 and T2 directly, not via the summary. The KVs carry non-textual information.

Setup B: idle past TTL, recomputes KV states on <sys> <summary> <T3>. But the fresh forward pass only ever sees the summary. T3's KVs are now computed in an alternate history where T1 and T2 never existed.

This puts the model in a weird OOD position of simulating what was happed to arrive at <summary> AND continue on to T3. Which makes the model worse.

We measured this in Memento:

43d5311

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr in other words, i always thought changing the prefix will invalidate the kv cache, for exactly this reason.. you are saying the cache remains even when prefixes change? what is the caching key?

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr wouldnt "tokens_turn_3” activations include also "pointers" to previous locations, which will messed up when masking?

43d22811

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr by "point" i mean "hold an address and refer to it", like pointers in a programming language. say the activations at layer 22 token 28 encode the information "the topic of interest is in token 12 layer 5". or "my child node is stored at token 15 layer 7".

Dimitris Papailiopoulos@DimitrisPapail

i am saying this

If the original session had any kind of context compression — compaction, sliding window masking, cache_edits pruning of old tool results, anything that drops tokens from the visible prefix while keeping later KVs — then the original KVs were computed in the presence of tokens that no longer appear in the rebuild input.

The cache rebuild after an idle session only sees what's currently in the session file. Whatever was compacted away is invisible to the recomputation.

43d3100

Dimitris Papailiopoulos@DimitrisPapail

you can test on open models which is what we did https://arxiv.org/abs/2604.09852 we did a fun experiment among many others where had a random number if turn n, masked after summary (that did not include random number) and tried to reconstruct form internal states of turn n+1 (after summary). Which was possivble far above random chance.

also saw accuracy drop a lot on tests when you do restarts vs not.

not sure how you could simulate on the api

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr how can i use claude's api to reach such a partial cache state?

43d9510

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr yes, i think as the soft embeddings as containing information, some of which is reference information that refer to past positions. there is evidence for this in the interpretability literature

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr Oh I see I’m not referring to actual pointers but the soft embeddings of KVs being flushed. I think we have an abstraction misalignment that’s confusing me :)

43d4710

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr in setup A, the cached T3 activations now point into positions in T1 and T2 that no longer exist, and even worse, are now replaced with vectors based on summary. why/how would that work?

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr

43d3600

Dimitris Papailiopoulos@DimitrisPapail

i am saying this

If the original session had any kind of context compression — compaction, sliding window masking, cache_edits pruning of old tool results, anything that drops tokens from the visible prefix while keeping later KVs — then the original KVs were computed in the presence of tokens that no longer appear in the rebuild input.

The cache rebuild after an idle session only sees what's currently in the session file. Whatever was compacted away is invisible to the recomputation.

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr what do you mean by point? i am confused.

43d3300

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr Oh I see I’m not referring to actual pointers but the soft embeddings of KVs being flushed. I think we have an abstraction misalignment that’s confusing me :)

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr by "point" i mean "hold an address and refer to it", like pointers in a programming language. say the activations at layer 22 token 28 encode the information "the topic of interest is in token 12 layer 5". or "my child node is stored at token 15 layer 7".

43d3100

Dimitris Papailiopoulos@DimitrisPapail

@yoavgo @Samhanknr what do you mean by point? i am confused.

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr in setup A, the cached T3 activations now point into positions in T1 and T2 that no longer exist, and even worse, are now replaced with vectors based on summary. why/how would that work?

43d2700

Zengineering@Samhanknr

@DimitrisPapail @yoavgo Do we know that this is how compaction works. In my naive understanding compaction wholesale replaces the conversation with a new summary. But codex does have a special compaction end point.

43d12

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr here is a recent example: https://belief.baulab.info/

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr yes, i think as the soft embeddings as containing information, some of which is reference information that refer to past positions. there is evidence for this in the interpretability literature

43d3410

(((ل()(ل() 'yoav))))👾@yoavgo

ok, if you train explicitly for this and only remove tokens for which there are summary tokens, that were created due to your training mechanism, then this is a smart idea and i see how it could work! but i thought you meant also for masking generic histories, which i still dont get

Dimitris Papailiopoulos@DimitrisPapail

you can test on open models which is what we did https://arxiv.org/abs/2604.09852 we did a fun experiment among many others where had a random number if turn n, masked after summary (that did not include random number) and tried to reconstruct form internal states of turn n+1 (after summary). Which was possivble far above random chance.

also saw accuracy drop a lot on tests when you do restarts vs not.

not sure how you could simulate on the api

43d5000

Dimitris Papailiopoulos@DimitrisPapail

@Samhanknr @yoavgo oh yes we dont know, but no matter how it works the point is when part of tokens are missing when you recompute KV states from the prefix, it hurts the model.

43d111

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr your description seem to assume the activations above a given token never refer back, and i doubt this is the case

(((ل()(ل() 'yoav))))👾@yoavgo

@DimitrisPapail @Samhanknr by "point" i mean "hold an address and refer to it", like pointers in a programming language. say the activations at layer 22 token 28 encode the information "the topic of interest is in token 12 layer 5". or "my child node is stored at token 15 layer 7".

43d900