Volatile scheduler dynamics and monthly updates drive this complexity
shower thought: AI inference engines like vLLM/SGLang might be the hardest “server”-shaped OSS to configure, ever made.
Like it’s probably harder to configure them optimally than even Postgres, Nginx, Kubernetes, maybe even Kafka/Zookeeper … just in terms of all the issues, bugs, crashes, scheduler dynamics, kernels etc and it changes every month
Many users call configuring inference engines like vLLM and SGLang difficult due to invisible flags and inherent complexity, while others praise the abstractions that enable quick speedups via simple arg changes.
No Digg Deeper questions have been answered for this story yet.
@ekzhang1 yes, but I was also surprised how well abstracted they are. many 4-5x speedups (non trivial) are just arg changes
@ekzhang1 Time to add vLLM to the Postgres ecosystem
@ekzhang1 It's because vLLM is basically a real-time scheduler pretending to be an API. One wrong tweak to `gpu_memory_utilization` and you either OOM at 3am or waste half your throughput.
@ekzhang1 a very "@bryan_gopal"-shaped solution in the works ;)
@ekzhang1 Not to mention the loading time with optimisations on every try!
@ekzhang1 Maybe because Postgres had time to develop and learn from previous database? I feel those inference engine add new model, paper and tricks every week with amazing speed but maybe less stabilisation?
@ekzhang1 😭 i might be lying to myself but really think this is due to inference's _inherent complexity_ to abstract serving workload on accelerators, still trying to see if we can make it easier and more stable
@ekzhang1 Why not use codex or something for configuring them
@ekzhang1 vllm took 15 min to load on 4H100 running DS4, which is not really bad!!!
@ekzhang1 laughs in Kubernates
@ekzhang1 @hsu_byron this is equivalent to 1970s & 1980s software
hw, sw, applications are all constantly evolving every week
postgres gets free perf upgrade every 2 years bcos of Moore’s law..
@ekzhang1 Java GC?
@ekzhang1 would be interesting to slap this in an autoresearch-style loop to bring to some hardware-optimal state autonomously
@ekzhang1 harder to configure optimally than sendmail?
@ekzhang1 The worst flags aren't wrong—they're invisible. `--max-num-seqs` silently OOMs your KV cache at long context, chunked prefill quietly eats decode tok/s. No log, no warning. You only find out by measuring. Postgres at least tells you what it picked.
@ekzhang1 In the world of artificial intelligence, nothing is simple. Challenges arise not only with software but also with hardware. However, thanks to companies like EXO Labs, it becomes a "piece of cake".
@ekzhang1 hard for humans
shower thought: AI inference engines like vLLM/SGLang might be the hardest “server”-shaped OSS to configure, ever made.
Like it’s probably harder to configure them optimally than even Postgres, Nginx, Kubernetes, maybe even Kafka/Zookeeper … just in terms of all the issues, bugs, crashes, scheduler dynamics, kernels etc and it changes every month
@ekzhang1 i totally agree. blindly updating versions without extensive perf testing can sign you up for a world of pain.
@ekzhang1 Hardware compatibility too.
I believe we’ll figure this out and be able to autotune the entire stack for you. It a big diff when optimized according to the workload
@ekzhang1 the complexity is purely because Postgres, Nginx, Kubernetes are all CPU bound and don't touch the GPU (as a core of their operation at least)