Model loading reportedly accounts for 55–70% of cold-start latency for small quantized LLMs on serverless CPUs
A post describing a Google paper says increasing Cloud Run memory from 4 GB to 8 GB gives the same model roughly twice the CPU, nearly halving warm inference time.
TLDR
According to a post describing a Google paper, model loading accounts for 55–70% of cold-start latency for small quantized LLMs on serverless CPUs. The post identifies getting model weights into memory—not generating tokens—as the frequent bottleneck. It also says increasing Cloud Run memory from 4 GB to 8 GB for the same model unlocks roughly twice the CPU, cutting warm inference time almost in half.
Combined views
—
1 Source, first seen ago
