LLMs reportedly decode two text streams in a single forward pass
HuggingPapers describes research in which averaging two texts’ numerical representations lets a language model predict both continuations at once.
TLDR
HuggingPapers says averaging the embeddings—numerical representations—of two texts lets a language model predict both continuations simultaneously. It describes this ability, called linear superposition, as built into the Transformer architecture: it fades during pretraining but can be restored with light fine-tuning. According to the summary, the researchers show how to decode both streams in a single forward pass through the model.
LLMs reportedly decode two text streams in a single forward pass
HuggingPapers describes research in which averaging two texts’ numerical representations lets a language model predict both continuations at once.
