THthebes@voooooogelTECH
unfortunately it's still not perfect, the main limitation imo being the reliance on the vocabulary. (same as the logit lens, naturally.) this already crops up a bunch in the paper, where they need to e.g. interpret tokens like "black" contextually as blackmail - which is easy enough in summitbridge, but not in full generality.
it also seems plausible to me that there are concepts that are difficult to verbalize in a single descriptive word/token, but yet are verbalized, just not by name. think vonnegut story shapes, or certain programming concepts. these might show up in the j-lens, but via tokens that wouldn't obviously name them to an uninitiated human - some promoted motor token or similar.
another way to look at it is what i'd want from an ideal (variable, not algorithm) technique, if i could wave a magic wand and get everything i want in one package, and where the existing techniques fail:
- not a closed vocabulary defined up front by the researcher: steering vectors / emotion probes fail this, you get streetlight effects, it's difficult to enumerate the relevant space in a principled way
- but not *overly* closed: single tokens are too restrictive. we want something like a library of concepts
- and with reliable labels: SAE autolabels range from OK to meh to outright misleading. SAEs are useful regardless, but i do feel more comfortable when the direction is label -> vector (like with steering vectors or the logit lens family) rather than trying to post-hoc slap a label on a complicated feature vector from a difficult to characterize process
it seems an even better technique is somewhere close at hand: maybe a library of concept vectors validated / pruned by verbalization? maybe SAE style merging on the j-lens to (carefully) combine and rename vocabulary entries into higher level concepts?
