MIT Researchers Study Dynamic Compression for RNNs
MIT researchers examine dynamic compression to let RNNs revisit and reorganize memory states during sequences.
Jyo Pari described in-context continual learning as requiring models to accumulate experience and reuse it later in the same sequence. Standard RNNs compress an ever-growing history into a fixed-size state that permits each token only a single write. The post examines dynamic compression, which would allow the model to revisit the past and reorganize its stored information. Ryan Bahlous-Boldi quoted the work, calling dynamic compression a prerequisite for full continual learning and thanking Jyo Pari for leading the project. Alex Zhang also shared the post.
In-context continual learning requires models to accumulate experience and reuse it later in the same sequence. But an RNN compresses an ever-growing history into a fixed-size state, where each token gets a single write into memory. We study dynamic compression: letting the…
Reactions from ranked influencers
3 postsContinual learning is soon going to be largely in-context. Our recent work brings forward something that is clearly a prerequisite to full continual learning: dynamically compressing information for future retrieval. Thanks to @jyo_pari for leading this awesome project!
In-context continual learning requires models to accumulate experience and reuse it later in the same sequence. But an RNN compresses an ever-growing history into a fixed-size state, where each token gets a single write into memory. We study dynamic compression: letting the…
In-context continual learning requires models to accumulate experience and reuse it later in the same sequence. But an RNN compresses an ever-growing history into a fixed-size state, where each token gets a single write into memory. We study dynamic compression: letting the…
Cool paper! Multiple passes over the sequence with fixed state models is a powerful axis!
In-context continual learning requires models to accumulate experience and reuse it later in the same sequence. But an RNN compresses an ever-growing history into a fixed-size state, where each token gets a single write into memory. We study dynamic compression: letting the…
