We validate our model framework by asking if we can recover existing interpretations of brain responses to language. We use our models to predict voxels that have shown tuning to either processing difficulty or meaning abstractness.

Check out the paper for WAY more details, analyses, and interpretations: https://arxiv.org/abs/2606.06857

🚨New preprint!🚨 We know that LM representations can be used to predict brain responses to language. But what *features* of these representations underlie this alignment? We use SAEs to find out!

@Michael_Lepori @threadreaderapp please #unroll

I had a ton of fun collaborating with @GretaTuckute and Kendrick Kay on this project! We hope that these close collaborations between mechanistic interpretability researchers and neuroscientists can continue to push both fields forward.

However, different regions differ in how strongly they rely on these features. For example, frontal regions tend to be more strongly predicted by surprisal than SAE/LM-based “content” features.

We find that SAE features used to predict a voxel in one region from one participant tend to generalize to other regions and/or other participants, indicating a largely shared feature basis!

We find that surprisal alone predicts processing difficulty voxels. In contrast, SAE features are required to predict meaning abstractness voxels. Further, we interpret the features, and find that they relate to aspects of concreteness (i.e., descriptions of scenery).

Three key results -Our models can interpret uncharacterized voxel populations -We find a shared feature basis in the fronto-temporal lang. network, with individual variation -Widely-used features for reconstructing LM representations are also useful for predicting brain responses

Extended thread: We introduce Augmented Sparse Encoding Models, a framework which uses SAEs to project LM hidden states into an interpretable basis AND includes sentence surprisal as a feature. We then learn a sparse linear mapping from this feature basis to brain responses.

Next, we identify and interpret a previously uncharacterized, but reliable, voxel population whose responses are predicted by "people-specific" features (i.e., descriptions of people doing things, relationships, pronouns).

These voxels 1⃣are not uniformly present across participants, demonstrating how we can investigate individual variability in linguistic meaning representations, and 2⃣largely reside outside the core fronto-temporal lang. network, but near areas associated with social cognition

We apply our model framework to a high-field 7T fMRI dataset of eight participants listening to 200 diverse sentences.

Thus, our framework recovers interpretations from prior work, indicating that it can provide accurate interpretations of brain responses.

Last but not least, we investigate the *properties* of the most predictive SAE features. Because we use Matryoshka SAEs, our feature basis is organized into bins that represent increasing granularity...

Finally, we investigate a larger set of voxels across the fronto-temporal language network to ask whether different brain regions and different participants rely on a shared feature basis during language comprehension.

...the first bin is small, and is broadly useful for reconstructing many LM representations (“most widely applicable features”), the next bin is contains more granular features (and so on).

@StartupYou @Michael_Lepori @StartupYou Hello, please find the unroll here: https://threadreaderapp.com/thread/2064429866100072557.html Have a good day. 🤖

@Michael_Lepori Fascinating direction. Moving from demonstrating alignment to understanding which features actually drive it feels like an important step toward more interpretable language and neuroscience models.