Allen AI's William Merrill finds hybrid transformer-RNN architectures outperform pure transformers on token-level content words

We found that the hybrid model advantage is highest on meaning-bearing words—the nouns, verbs, & adjectives that say what a sentence is about.

On function words like "the," "of," & "is," its advantage over transformers remains, but is more muted.

Hybrid (transformer–RNN) models are fast becoming a serious alternative to the transformer, but a big question remains: how do they process tokens differently & how does this impact performance?

We compared our transformer (Olmo 3) & hybrid (Olmo Hybrid) models to find out. 🧵

amazing work as usual from @YanhongLi2062 & @lambdaviking

We find that filtering loss by token type is a promising way to spot architecture differences.

Overall loss (left) makes transformers & RNNs appear even, with hybrids ahead. Filtered losses (center, right) reveal where transformers & RNNs each shine.

We hope this kind of token-level comparison helps the field build better, more architecturally diverse models. Going forward, we plan to explore applications of these findings to pretraining evals.

✍️ Blog: https://allenai.org/blog/hybrid-token-prediction 📄 Tech report: https://arxiv.org/abs/2606.20936

In contrast, the transformer matches the hybrid model when the next token completes an n-gram repeated verbatim from earlier in the passage.

Transformers also match or beat hybrid models at predicting closing brackets like the ], but not opening brackets like [.

A transformer’s attention layers can look back at any earlier token exactly.

A hybrid model swaps most of those layers for recurrent ones that excel at sequential processing.

Do these differences give hybrid models and transformers different strengths?

To find out, we scored how well Olmo 3 & Olmo Hybrid predicted different kinds of next tokens across articles, books, papers, code, HTML, & LaTeX.

The models are matched on data, tokenizer, & training recipe, so a gap in their predictions points to architecture differences.

link to paper https://arxiv.org/abs/2606.20936

@YanhongLi2062 Very fulfilling to play around with part-of-speech tags and n-grams

@allen_ai It'll be interesting to see if the advantage of hybrid models working better on meaning-bearing tokens carries over to downstream reasoning and agent tasks, where semantic understanding often matters more than next-token accuracy.

@allen_ai attention is not all we need anymore

@allen_ai the honest answer is that the distinction between transformers and hybrids isn't about the architecture, but about what they can handle. if you're doing something unusual with your tokens, a hybrid might just be the better choice

@lambdaviking @YanhongLi2062 good to see hybrids actually showing teeth on content words instead of just copying patterns