@francoisfleuret Except for the many real cases where you don't know yet that something will be important until you see something in the future. RNN is toast, Transformer handles it trivially.

My intuition is opposite - that conscious thought is primarily from sparse activations roughly corresponding to distinct concepts that basically get looked up through learned associations with other such concepts (symbolic reasoning-like). I'm imagining the RNN "strong conceptual embeddings" as fairly dense and opaque, which I would more think of as unconscious or intuitive.

@giffmana @francoisfleuret Except if we accept we have tools and can behave like humans by doing search over the past only when necessary and in a principled manner: https://arxiv.org/html/2510.14826v1

Best of both worlds: no infinite KV Cache and yet everything is still retrievable.

@francoisfleuret Conceptually, without literally remembering everything, they can condense, memorize and retrieve all the ideas and concepts they have been exposed to. Expensive to train today, in the future your "personal database" will be one just continually/continuously trained. You 2nd brain

@francoisfleuret Aren't there a few communities trying to do that, for quite some time?

@francoisfleuret Hotter take: I’m running a cognition substrate that doesn’t use tokens at all.

155+ days continuous operation, learning through thermodynamic state evolution no transformers, no RNNs, no resets.

https://zenodo.org/records/19703134

@francoisfleuret RNNs over Transformers in the big 2026???

@giffmana @francoisfleuret Desire to “fight” something seemingly simple is not new, there were lots of attempts to eg create something other than transistor or even something other than using binary in computer, some attempts even looked like they might work, eg ternary based: https://en.wikipedia.org/wiki/Setun

@tak3sh8 Yes.

@francoisfleuret Wasn't the first part basically the point of "Hopfield Networks are All You Need"?

@francoisfleuret Ideally an architecture they utilises the pros of attention and RNNs & derivatives, hell, even the cons of RNNs could be really good

@francoisfleuret Hotter take :) Context compaction makes our current agents into RNN-transformer hybrid systems

@francoisfleuret @mike64_t https://goombalab.github.io/blog/2025/tradeoffs/

@francoisfleuret Reading theses two paper given valuable information, and prove some advantage of DPLR RNN over trarnsformer reading expressivity

https://arxiv.org/abs/2311.00208v3

https://arxiv.org/abs/2603.03612

@francoisfleuret @mike64_t Transformers in a loop are much stronger, no?

@francoisfleuret You would have loved @tserre keynote at CVPR

@francoisfleuret H-net (Albert Gu) shows that SSMs are strictly better at compressing information and generating relevant chunks of tokens that transformers. Papers analysising correlation between models and the brain also show that SSM correlate better with brain activations that transformers

@giffmana @francoisfleuret That’s for known RNN architectures I believe, we did not find an way to model and scale long range dependencies in RNN but maybe one day we will

@francoisfleuret And this is because RNNs have to compress everything they've seen into a single hidden state?

@francoisfleuret This has been my theory, too. Unfortunately it's really difficult to get good state space representations. Sliding window attention can be seen as a state space representation where you learn to reach further back via stacked sliding windows and empirically that falls off