Announcement
Zyphra introduces hierarchical attention as a path to subquadratic scaling
ZHA searches a learned tree of compressed summaries; Zyphra projects a 35× speedup over dense attention at 10 million tokens.
TLDR
Zyphra describes its method as a path to subquadratic attention, where processing cost grows more slowly than input length squared. It projects a 5× speedup over MSA at 10 million tokens. At 1 billion tokens, it projects roughly 400× over dense attention and 55× over MSA.
Combined views
111.4K
16 Sources, first seen ago
