My METR colleague @slimshetty_ has an interesting post exploring the nature of improvements to NanoGPT Speedruns over time. I find the history of speedruns in general fascinating.
One thing that stands out is the cumulative effect of relatively shallow contributions.