Swyx, creator of ai.engineer, argues slow context window growth cannot replace model weight updates for continual learning

"context length is the slowest Moore's Law"

http://AI.Engineer curator swyx says context windows grow too slowly to replace weight updates for AI memory.

"We've gone from a thousand token context to a million in three years, which is actually kind of slow."

"You can't rely on infinite context because we don't have infinite context because we don't have infinite memory."

@swyx