Modal Servers Deliver 6x Faster Inference with Auto Endpoints

Modal Web Functions aren't deprecated by the way -- they just solve a different problem.

I like to think of it like the difference between TCP and UDP.

I'm not a modal shill because I work there. I work there because I'm a modal shill.

TCP is cozy and solves key problems every application needs to solve, but in a generic way.

UDP rips, but like a chainsaw, you have to make sure it doesn't rip you!

Read the blog here: https://modal.com/blog/serverless-servers.

If you do, I hope some of that awe rubs off on you and inspires you to add to the infrastructure we all rely on, innumerable times every day.

That "Routing Plane" box hides a lot of complexity, just like Modal Servers hide complexity for people deploying inference services.

This blog post lifts the lid on that black box. We hope it's useful to other folks building proxy/routing systems at scale.

For @modal endpoints, we had to build our own routing and proxy layer.

We're making this primitive generally available as `app.server()` — deploy any server code for low-latency, high throughput serving. Same fast cold start times.

It's still so early

This system is, to me, literally awesome. I am in awe at the power of the infrastructure we build on, from chips and their NICs to the Rust programming language that pushes them to their limit, fearlessly.