Reaction
Copy-Paste Language Models aim to improve token copying
The announcement claims gains in downstream accuracy, long-context performance and retrieval-augmented generation memory efficiency at under 3% overhead.
TLDR
A post introducing Copy-Paste Language Models says they learn to copy tokens better, improving downstream accuracy, chain-of-thought and long-context performance, and memory efficiency for retrieval-augmented generation at under 3% overhead. It also claims they partly mitigate the LM head bottleneck. A separate commentator says CPLMs allow a KV sliding window without performance loss and mitigate much of the softmax learning bottleneck.
Combined views
807
2 Sources, first seen ago
