—
Not ranked yet
—
Not ranked yet
@NeelNanda5@BlancheMinerva @afg9000 @OrionJohnston We're talking about parameters, right? Those are quadratic
@BlancheMinerva@NeelNanda5 @OrionJohnston I guess it depends on where / how you think about adding the additional neurons. I was imagining adding a neuron as adding 2 d_model parameters. In some regimes that comes out quadratic but in others (e.g., adding an additional layer to the model) it doesn’t.@NeelNanda5@BlancheMinerva @afg9000 @OrionJohnston We're talking about parameters, right? Those are quadratic@BlancheMinerva@NeelNanda5 @OrionJohnston I guess it depends on where / how you think about adding the additional neurons. I was imagining adding a neuron as adding 2 d_model parameters. In some regimes that comes out quadratic but in others (e.g., adding an additional layer to the model) it doesn’t.@NeelNanda5@BlancheMinerva @afg9000 @OrionJohnston We're talking about parameters, right? Those are quadratic@BlancheMinerva@NeelNanda5 @OrionJohnston I guess it depends on where / how you think about adding the additional neurons. I was imagining adding a neuron as adding 2 d_model parameters. In some regimes that comes out quadratic but in others (e.g., adding an additional layer to the model) it doesn’t.@NeelNanda5@BlancheMinerva @afg9000 @OrionJohnston We're talking about parameters, right? Those are quadratic@BlancheMinerva@NeelNanda5 @OrionJohnston I guess it depends on where / how you think about adding the additional neurons. I was imagining adding a neuron as adding 2 d_model parameters. In some regimes that comes out quadratic but in others (e.g., adding an additional layer to the model) it doesn’t.
