Announcement

Shared-cache transformer training tweak reportedly removes 81–89% of the primary continuation penalty

The researcher says generation had higher loss than prefill, even with ground-truth token inputs.

6 Sources, 20d ago, first seen 20d ago

Architecture of the default tied transformer: one untied prologue block, four distinct core blocks reused for three iterations with an LSTM-style carry, then two untied epilogue blocks. The core reads one common donor KV cache for preceding tokens, produced by its final block on its final iteration. Each block retains its own current-token KV. There are seven block weight sets and fifteen block applications. With separate prologue and epilogue caches, sharing reduces persistent cache sets from fifteen to four.

TLDR

After leaving Google DeepMind, a researcher describes a short, compute-constrained project on a transformer with one shared core key-value cache. They say their two-pass training still produced cache histories that differed from those during continuation. Matching the passes’ donor keys and values during the final 10% of training removed 81–89% of the primary continuation penalty at three tested widths, they report. The token budget stayed the same, while full-cache loss rose by 0.0011–0.0025 nats.

Today's Rank

—

Not ranked yet

Combined views

9.6K

6 Sources, first seen 20d ago

Shared-cache transformer training tweak reportedly removes 81–89% of the primary continuation penalty

TLDR

Combined views

6 Sources

Combined views

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

Sentiment

Summary

6 Sources

6 Sources

6 Sources