Users praised the preprint showing ESM-2 and ESM-3 protein models capture similar biology because they called the finding cool and the summarization great.
No Digg Deeper questions have been answered for this story yet.
Surprisingly, one attention head (L0H7) does most of the work for structure token reasoning
Ablating it alone changes 40% of secondary structure predictions. 10 random layer-0 heads: <17%.
Also happy to hear that @KestenGal observed similarly disproportionate contribution of L0H7 in their article about protein repeats https://arxiv.org/abs/2602.23179
Very excited to share this first exploration into mechanistic interpretability of these fascinating language models -- and looking forward to doing more of this type of work! https://www.biorxiv.org/content/10.64898/2026.05.12.724593v1
In summary, two architecturally distinct models that operate across different modalities converge on a shared biological vocabulary
Structure tokens sharpen that vocabulary rather than rewriting it
Steering vectors, attribution patching, and sparse feature circuits all mechanistically confirm that these features sit within the model's causal pathway
This suggests that ESM-2 already infers structural regularities from sequence alone
Structure tokens don't rewrite the vocabulary. They sharpen representations that the sequence context has already approximated
Despite structure tokens representing an input modality unique to ESM-3, they don't create a new feature vocabulary
The 15.2% of features most activated by structure tokens are more convergent with sequence-only ESM-2 than invariant features (r=0.54 vs 0.45).
And the features both models agree on are the ones that matter.
Convergent features → AUROC 0.925 on functional site detection. Architecture-unique features → 0.661.
Shared = signal. Unique = mostly noise.
@jlsteenwyk @KestenGal Wow this is a very cool finding
@jlsteenwyk Really nice work, and a great summarization of your finding!
@ChrisHayduk @KestenGal :)
@DdelAlamo Thank you! :D