Researcher Lists Key Questions On AI Capabilities By 2029
Source Posts on X
Cluster sources
some questions i'd ask if i could talk to someone from 2029 about ai: 1. are we still in the paradigm where everyone uses frontier models or does it make sense now for most people to use weaker models and only a small % of users to use frontier models (i.e. scientists working at national labs, people developing GPU kernels). 2. can frontier models generate at 1 million tokens per second yet? 3. do frontier models have a 1 billion token context window? 4. are we still saturating benchmarks at the same speed as today, or is it faster/slower? 5. is most of the data we train on synthetic or natural? is the synthetic data fully synthetic or is it computer-verified (i.e. compiler or Lean proof checker or...)
my predictions: 1. unpopular prediction but i think everyone will always wanna be at the frontier and most people won't use non-frontier models. 2. i think by 2029 we'll be able to generate at 100k tok/second with smaller models, my prediction for frontier models is 2k tokens/sec 3. there's a huge difference between a theoretical context window and a practical *usable* context window. i think by 2029 we'll have models that claim to have a 1b context window but frontier models will probably have an effective context window closer to 50m tokens 4. this is really hard to quantify, but i think that by 2029 we'll have a harder time making our models better, the overall rate of improvement in general abilities in the models will not be as quick as it is today. 5. this question is also really hard to quantify. i think most data we train on will be *based* on real-world data found in the wild (as in swe-bench), but it might be augmented/validated by deterministic formal tools like compilers or proof checkers
pic in first post by my brother @ori_press