Report
Hyper-Connection Factory brings 11 LLM residual-connection variants into one open-source repo
Its creators say the variants use a shared transformer body and one training recipe across models from 38 million to 1.2 billion parameters.
TLDR
The creators of Hyper-Connection Factory say their open-source repo brings together 11 residual-connection variants for large language models. They say they used a shared transformer body and one training recipe for models ranging from 38 million to 1.2 billion parameters, at roughly 20 tokens per parameter.
Combined views
1 Source, first seen ago
