A really cool idea! The question of how we can train sequence models such that they remember things that are T timesteps in the past without backpropping through T timesteps remains one of the core problems in ML, and this looks like an inspiring approach!

We introduce a method for training RNNs that is time-parallel and does not suffer from vanishing/exploding gradients.

Key idea is to decouple learning 1) what should be remembered (can be done without recurrence) and 2) how to update memory (can be one-step supervised by #1).

It's so over. The kids are now giving their neural networks DMT.

Jokes aside, this is a clever + promising way for time-parallel training of sequence models—one of the original motivations for the transformer. Despite being an alternative solution, it is also complementary.

Listened to @phillip_isola talking about this work. Really clever idea!

@akarshkumar0101 Awesome stuff. We also showed that you can pre-train a RNN without recurrence by using the transformer backbone to forecast latent states and training the RNN on one-step latent predictions in Next-Latent Prediction Transformers (https://arxiv.org/abs/2511.05963)

SMT is akin to off-policy behavior cloning, and is mainly for pretraining.

To stabilize RNN rollouts, we introduce an on-policy imitation algo: DAgger Memory Training (DMT), a relatively lightweight fine-tuning phase.

Long Range Memory

Encoder+decoder are Transformers and can lookup any token in the past and future and associate them immediately via attention (O(1) gradient path).

This solves vanishing gradients (left).

With this, SMT can learn long-range memory and even train next-pixel prediction RNNs (right).

Time-parallelism

SMT is fully time-parallel, making it efficient on GPUs.

SMT outperforms BPTT in sequential computation required to achieve a certain loss.

In scaling laws, the y-axis is often loss. But what if it was instead compression?

In SMT, increasing training compute allows you to get to the same loss, but with a smaller memory state size.

This is a new way to spend your compute.

Thanks to @phillip_isola for inspiring me to pursue this direction in depth and providing invaluable guidance!

SMT estimates the oracle via a time-parallel encoder trained to embed past context into a representation that a decoder can use to predict the future.

This creates memory states that remember important info and purposefully forget unimportant details, similar to biological memory.

SMT+DMT are a fundamental improvement over BPTT because they perform credit assignment across a sequence in a qualitatively different way (without recurrence).

Check out the paper for many more experiments and insights.

@vincesitzmann Thanks Vincent!

What should be remembered: a compressed representation of the past that predicts the future (predictive state).

How to update memory: predict the next predictive state.

@akarshkumar0101 I like anything getting us off of BPTT.. but.. what if the oracle doesnt exist. what if we are trying to solve a class of problems humans dont know how to solve. then there is no trace to train on. thats what we have to solve.

@MinqiJiang Thanks Minqi!

@akarshkumar0101 cc @neurallambda

@vincesitzmann For AR we use embeddings; for diffusion we use encoders/decoders. Yet for hybrid AR-diffusion models like recent world models, we know too little about what makes a good encoder.

@danfei_xu @phillip_isola this work is actually valid but does smt really hold up against transformers in the long run anyway

@FrancoisChauba1 @akarshkumar0101 There is a mechanism for reconstruction although how to implement it that's a different matter. It is the experiments section.

https://github.com/Lexlangel/Interaction-dynamics-core/tree/main