• Home
  • Technology
  • Gaming
  • Entertainment
  • World & Business
  • Science
  • Sports
  • AI
HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • HomeTechnologyGamingEntertainmentWorld & BusinessScienceSportsAI
    • Home
    • Technology
    • Gaming
    • Entertainment
    • World & Business
    • Science
    • Sports
    • AI
    AI
    Announcement

    Spotlight's claimed linear computational cost and a potentially large constant factor

    A user says each access touches nine cells, each a full d_k×d_v matrix per head.

    EV
    T(
    AP
    6 Sources, ,

    TLDR

    A post quotes a claim that Spotlight combines attention's growing memory and capacity with the linear computational cost of fixed-state recurrent models. The poster says the constant factor seems large: each access touches nine cells, with each cell a full d_k×d_v matrix per head.

    Combined views

    8.3K

    6 Sources, first seen 6h ago

    Combined views

    8.3K

    6 Sources, first seen 6h ago

    73 likes
    Today's Rank

    —

    Not ranked yet

    Today's Rank

    —

    Not ranked yet

    6h ago
    first seen 6h ago
    73 likes
    10 comments
    45 saves
    35 reposts

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    Featured Source
    10 comments
    45 saves
    35 reposts

    6 Sources

    @apjacob031/ What would it take for LLMs to be able to grow their own capabilities? We introduce spotlight, a memory architecture with growing capacity but with linear computational complexity. We show that this suffices to host evolving ecosystems like software by building Python directly into an LLM. We also share some very early language modeling results where it outperforms attention and linear attention on long context capabilities!6h
    @teortaxesTex«Spotlight combines the growing memory and capacity of attention with the linear computational cost of fixed-state recurrent models» The constant factor seems large though (an access touches 9 cells; a cell is a full d_k×d_v matrix per head) but I'd expect that from CL design?1h
    @EugeneVinitskyLinear complexity and growing memory; another wacky exciting result from @percepta1h

    Sentiment

    Positive——Negative

    Summary

    Not enough discussion yet.

    No sentiment analysis available yet.

    6 Sources

    @apjacob031/ What would it take for LLMs to be able to grow their own capabilities? We introduce spotlight, a memory architecture with growing capacity but with linear computational complexity. We show that this suffices to host evolving ecosystems like software by building Python directly into an LLM. We also share some very early language modeling results where it outperforms attention and linear attention on long context capabilities!6h
    @teortaxesTex«Spotlight combines the growing memory and capacity of attention with the linear computational cost of fixed-state recurrent models» The constant factor seems large though (an access touches 9 cells; a cell is a full d_k×d_v matrix per head) but I'd expect that from CL design?1h
    @EugeneVinitskyLinear complexity and growing memory; another wacky exciting result from @percepta1h