The model was trained on pre-1931 books over two days.
I ran a quick 32k context extension on Talkie 1930 using pre-1931 books and YaRN. It's probably possible to squeeze some more long context performance from Talkie, but for two afternoons on a single A100 node I think this is reasonable, given where the model started.
Pretty impressive it was able to answer questions about @paulg s sandwiches with only pre-1931 knowledge.
I ran a quick 32k context extension on Talkie 1930 using pre-1931 books and YaRN. It's probably possible to squeeze some more long context performance from Talkie, but for two afternoons on a single A100 node I think this is reasonable, given where the model started.
The model was trained on pre-1931 books over two days.
I ran a quick 32k context extension on Talkie 1930 using pre-1931 books and YaRN. It's probably possible to squeeze some more long context performance from Talkie, but for two afternoons on a single A100 node I think this is reasonable, given where the model started.