Adrien Ecoffet, OpenAI research scientist, warns that loosely applying recurrence to neuralese blurs RNN and transformer distinctions and may discourage LSTM research
Ryan Greenblatt suggests describing it as opaque serial depth with layer reuse.
@AdrienLE I'm most worried about opaque serial depth without legible token bottlenecks. As in, opaque cognition (under reasonable elicitation). Thus, LSTMs are scary.
I think "recurrence" is intended to include "passing activations from the last layer to the first" (including soft tokens)
Adrien Ecoffet@AdrienLEIt bothers me that people are increasingly using the word “recurrence” to refer to neuralese. It is possible for an RNN to have legible CoTs or for a transformer to think in neuralese. Aside from betraying technical confusion, this stigmatizes harmless research programs (“reinvestigate LSTMs”) while normalizing risky ones (“train CoTs to use soft tokens”)
@AdrienLE I don't feel good about CoT if the model can easily not use it in principle.
BTW, I'm maybe sympathetic to avoiding the term "recurrence" and instead saying something like "total opaque serial depth without a legible bottleneck and with layer reuse".
Ryan Greenblatt@RyanPGreenblatt@AdrienLE I'm most worried about opaque serial depth without legible token bottlenecks. As in, opaque cognition (under reasonable elicitation). Thus, LSTMs are scary. I think "recurrence" is intended to include "passing activations from the last layer to the first" (including soft tokens)