Report
@ShivamDuggal4 introduces Latent Lattice Language Models: computation goes sideways
@ShivamDuggal4 says the architecture adds a horizontal lattice of latents alongside the usual vertical processing stream.
TLDR
According to @ShivamDuggal4, Latent Lattice Language Models add a horizontal computational axis alongside the vertical residual stream used to process tokens. @JohnCLangford describes a hybrid of transformers and parallel-scan linear state-space models, and says it improves model quality in comparisons matched for parameters, tokens or compute.
Combined views
782
4 Sources, first seen ago

Comments