Fwiw - I understand that this is the concensus view, but I think history will look back with surprise that it didn't bear out in the end.
In the 1960s, an employee at IBM or Bell Labs would have said the same thing about the Mainframe computer... and they were incredible (and many are still in use today).
But it wasn't just "bigger mainframes forever" anymore than the library was just "bigger library of alexandria forever".
I say that as someone who joined DeepMind nearly 10 years ago doing language modeling research. I have had access to large scale and small scale compute during that time.
I personally think there's an enormous amount of low-hanging fruit which doesn't require.
The future is networks of neural networks: - better routers - better benchmarks - better access to non-public / niche information - better pricing mechanisms - better source attribution - better unlearning - ...
There's so much great research to be done. And much of it remains low-hanging because there are some subtle reasons why highly resourced orgs don't tend to pursue them.
If you want to work on pretraining-for-AGI, join OpenAI, Google, Meta or the Anthropic/XAI/Cursor supergroup.
The bitter truth of the widening compute gap is that all the problems which are actually on the critical path to AGI now demand that level of compute.
