Reaction
mHC reportedly saturates at 24 layers in an experimental 400M model
A commenter suggests mHC is likely at least no worse than AttnRes for the very wide, shallow model design targeted by DSV4.1-Flash. They would like to see internal lab ablations.
TLDR
A commenter says mHC saturates at 24 layers in an experimental 400M model. They suggest it is likely at least no worse than AttnRes in the very wide, shallow model setting targeted by DSV4.1-Flash, and would like to see internal lab ablations.
Combined views
3.2K
2 Sources, first seen 4h ago
likes