Prime Intellect ML engineer Elie Bakouch finds universal layer geometry across 38 unrelated open-source language models
The analysis used Jacobian-lens similarity to map hidden layer alignments.
Story Brief
The analysis used Jacobian-lens similarity to map hidden layer alignments.
Related Stories
2 linkedneuronpedia/jacobian-lens at mainWe’re on a journey to advance and democratize artificial intelligence through open source and open science.3h agohuggingface
J-lens CKA explorerAt every layer, each vocabulary token has a steering direction — its J-lens vector, the direction in activation space that pushes the model toward eventually saying that token. Each cell of the matrix asks: do two layers arrange those 4,096 directions the same way? 1.0 = identical geometry, 0 = unrelated.3h agoeliebak.com
J-lens CKA explorerAt every layer, each vocabulary token has a steering direction — its J-lens vector, the direction in activation space that pushes the model toward eventually saying that token. Each cell of the matrix asks: do two layers arrange those 4,096 directions the same way? 1.0 = identical geometry, 0 = unrelated.3h agoeliebak.comCommentary on X
Highest ranked
Irving Ernesto@ieqr_TECH
So this feeds the theory that every model is kind of converging into the same distribution, really interesting results!! That they open sourced this paper is a true gift, I've been doing quite a lot of experiments (trying to find a better abliteration) and it's just a goldmine of findings
