Commentary on X
Inkling small is ~3.5x smaller but the performance looks insane? I think the comparison table here gives quite abit of insight as to what requires larger model capacity (things like TerminalBench and SimpleQA) while "reasoning" can be condensed to some "core capability"
4:54 PM · Jul 15, 2026Attention wise: 1) Sliding window is kind of surprising since they have the queen of linear attention on the team. 2) Convolutions (basic 1d conv) on KV and residual. (I feel like someone else has done this before but I don't remember atm) 3) No RoPE. Attention is basically - states = proj(hidden) - For each query token, project to get relative extent of mixing. - position_bias = gather relative extent based on distance - attn_weights + position_bias Inspired by https://arxiv.org/abs/1809.04281 (not familiar with this but Vaswani and Shazeer on it too)
4:54 PM · Jul 15, 2026Combined views
1.9K
2 posts, first seen 6h ago