Amiri Hayes along with MIT CSAIL colleagues has built a pipeline that turns selected attention heads inside models like GPT-2 and Llama-3B into short, executable Python functions. The programs are synthesized from observed attention patterns on datasets such as TinyStories, then swapped directly back into the transformer stack.
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
Hybrid models that replace up to a quarter of attention heads show only modest perplexity rises and hold steady on standard QA benchmarks. The work does not claim every head can be replaced or that the approach scales unchanged to frontier models.
Because the replacements are ordinary Python rather than post-hoc descriptions, researchers can now read, edit, or verify what individual heads compute. The paper releases the full library of 1,664 programs and the synthesis code for others to test.
Users are excited about researchers replacing transformer attention heads with readable Python programs, praising the work as very cool and a big mechanistic interpretability win.
No Digg Deeper questions have been answered for this story yet.
👉 New preprint! Automated interpretability by approximating / replacing NN components (here attention heads) with programs.
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@amirihayes_ @VictorTaelin
@amirihayes_ Very cool work, this paper might be related and interesting for you (very similar themes as yours): https://arxiv.org/abs/2602.08857
Very cool work ⬇️
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@amirihayes_ https://github.com/AmiriHayes/explaining_attention_heads/blob/main/data%2Fgpt2_programs.py
Explaining Attention with Program Synthesis asks a sharp question: what are attention heads actually doing, and can we describe it precisely?
- The paper uses program synthesis to automatically generate symbolic descriptions of attention head behavior, moving beyond heatmaps toward interpretable rules. - Synthesized programs capture the computational patterns of individual heads in a human-readable form, making mechanistic claims testable. - The approach surfaces structured, repeatable behaviors rather than treating attention as a black box of soft weights.
Lines of inquiry it opens: - Can symbolic mechanisms improve transformer compositional abilities? - Do transformers learn generalizable algorithms or instance-based patterns? - Why do standard transformers fail on problems requiring serial algorithmic reasoning? https://inquiringlines.com/related/2606-19317-explaining-attention-with-program-synthesis/
@amirihayes_ wow this seems wild! taking a look!
@amirihayes_ very cool!
@amirihayes_ Everything is pattern recognition.
@amirihayes_ Wow. Neat. Big mech int win
New paper! We introduce a new automated interpretability technique where attention heads are explained with Python programs. Turns out you can drop-in replace ~40% of attention patterns in Llama-3B with outputs of these programs and barely affect task performance!
More broadly, I’m excited about the actionable implications of this technique: Understanding attention phenomena has historically led to architectural improvements in Transformers (see e.g. attention sinks), and I’m excited about the potential for this technique to uncover more such opportunities.
Make sure to check out @amirihayes_ thread below! ⬇️
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@pplank8 @amirihayes_ thanks for pinging me here!!
@amirihayes_ @AbhishekAs34298
@amirihayes_ @fchollet
Very cool work!
What if attention were code? We show that many attention heads in transformer LMs can be replaced by human-readable Python programs. Swap them in and the model barely notices.
See our experiments here: Explaining Attention with Program Synthesis [https://arxiv.org/abs/2606.19317]
@amirihayes_ very cool
@amirihayes_ great work!
