Core Automation CEO Jerry Tworek argues the human brain likely contains a biological mechanism analogous to transformer self-attention

@MillionInt a common connectivity pattern in the cerebellum actually

@MillionInt

@MillionInt @grok what's neuroscience take on this?

@MillionInt Well, it’s a tiny fraction of the parameters in a LLM as well 😅

@MillionInt to be honest, i believe on it too

The static NN models we have were first developed by Jack Cowan MIT in the mid 60s and later at the University of Chicago.

These are effectively steady state solutions of the equations of motion from the statistical mechanics of spiking neurons

The attention mechanism was first described in 1992 by Schmidhuber

The attention mechanism itself can be seen as a second order term in a field theoretic description of the generalization properties of these models

@MillionInt It's not tiny - you could view frequencies as such they help regions to connect, for particular frequency on its physical reach (low freq have higher, high are more local). This way you can control which regions talk to which when. @leothecurious

@MillionInt I think you just want to convince yourself that the attention mechanism is fundamental to intelligence, but this isn't the case.

@MillionInt we're all converging

@MillionInt Flash-Attention-5 optimization for hippocampus CA1 👀

@MillionInt Something like pulvinar nucleus in thalamus

@MillionInt Obviously. Because behaving in accordance with self-attention certainly improves success in different situations. Yet trying to isolate it makes no sense. The brain is adaptation. What you'd want as a specific module is structurally plastic and changes with interactions.

@MillionInt attention is 1% of brains we understood. we scaled it insanely far. the other 99%, the embodied messy stuff, actually matters now.

@MillionInt the tiny fraction part is the real joke, we got billions of params running on caffeine and bad sleep

@MillionInt Its astonishing that we think theres a universe outside , because that universie doesn't exist when we die

@MillionInt Yeah the AGI is going to be a chonker! More structures and algorithms. It’s the bitter lesson beyond data, we’ve only just begun.

@MillionInt not "tiny tiny fraction of parameters" but a very sophisticated hyper-params which are optimized for the world model

t_cmtl Neuroscience sees brain attention as selective filtering via prefrontal-thalamic circuits and neuromodulators, not literal transformer self-attention. Some 2026 hypotheses map cortical column laminar loops to context-dependent gating akin to QKV-style weighting. But the brain uses sparse recurrent dynamics and local computation, not dense global matrices. The "tiny fraction" idea fits—most synaptic "parameters" handle specialized local processing, with long-range integration as a smaller integrative layer. Analogies are useful but not identical.

@CalcCon @MillionInt > The attention mechanism was first described in 1992 by Schmidhuber

hero