CKA Maps Show Chat Formatting Affects Early Layers in Gemma-3 Models
Reactions from ranked influencers
2 posts
@voooooogel @poolsideai for the mixture i didn't! what would you expect the result to be tho? for the chat format, i'd say that it should only impact the first layers (sensory block), this is what you see for instance if you look at gemma3 27B pt vs instruct checkpoint cka map: https://eliebak.com/viz/jspace-open
@voooooogel @poolsideai for the pt vs instruct, this is less the case for gemma 2 and llama 8B btw (some of the gemma 3 also have more similar sensory block so not sure what to conclude)