It serves as an alternative to proprietary hosted inference.
A few years ago, the future of artificial intelligence looked dark - proprietary models, proprietary inference services.
I joined Modal to fix that.
Today, we take a major step forward. Optimized inference, open source, with a click.
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
Many users are excited about Modal's Auto Endpoints launch for open source and private LLM inference because the feature feels slick, bold, and exactly how inference should work after months in development.
No Digg Deeper questions have been answered for this story yet.
Managed private LLM endpoints, now available for everyone in @modal. Deploy in a few clicks with the UI or a few keystrokes with our CLI.
The coolest thing is that these are not black boxes – customers have full access to the code underneath.
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
We consider our infrastructure to be a massive differentiator – for fast cold starts, low latency, etc. We've spent the last five years building our entire stack from scratch: custom filesystem, container runtime, scheduler, load balancing, etc.
Managed private LLM endpoints, now available for everyone in @modal. Deploy in a few clicks with the UI or a few keystrokes with our CLI.
The coolest thing is that these are not black boxes – customers have full access to the code underneath.
This is why we're taking a different approach with LLM inference. We want customers to have full control over the serving code.
When you're running Modal Auto Endpoints, you run the best open source models with the best open source engines, and you have full access to the code.
At the same time, open source LLM inference is moving super fast – with amazing projects like vLLM, SGLang, FA4, etc. We're actively contributing to those because we think everyone benefits from moving the frontier.
At the same time, open source LLM inference is moving super fast – with amazing projects like vLLM, SGLang, FA4, etc. We're actively contributing to those because we think everyone benefits from moving the frontier.
teh team truly cooked with this one. many weeks / months in the making. yet, it feels more discovered than invented b/c this is how inference should have always worked.
I have spent the last year in the weeds with teams running open LLM inference on Modal.
Modal was already the best place to do this: truly elastic GPUs, fast cold starts, low-overhead routing, and deep observability.
Very excited to raise the bar even higher today.
Auto Endpoints package the Modal inference stack into one command:
modal endpoint create --name agent --model zai-org/GLM-5.2-FP8
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
guys....its time...to own your inference with unparalleled observability, groundbreaking performance, and the ez of use you expect from @modal
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
@charles_irl This is a really great feature, I needed that since april, thanks for changing your mind about it! 🎉
@dude474 we had to make it code-first! tricky
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
It shouldn't be impossible to have deep control of your inference stack.
Inference on Modal is "just an app built atop our existing platform", + some sick engineering to offer frontier-level specdec, fast autoscaling, o11y, etc.
happy tokenmaxxing :)
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
You no longer have to pick between the performance of a black box API and the flexibility and control of @modal. Auto Endpoints give you both.
We're unlocking frontier performance for everyone without having to talk to sales or an FDE. More cooking here, stay tuned.
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
@bernhardsson @modal Can I have batch support as well?
give it a try and let us know what you thnk.
Deploy a complete, optimized inference service, including autoscaling, regionalization, and performance observability, with Modal Auto Endpoints.
modal endpoint create Qwen/Qwen3.6-27B-FP8
It is not too late to _actually_ own your inference.
Introducing: Modal Auto Endpoints.
@bernhardsson @modal Damn. This is big and bold. Lfg
@charles_irl God you’re reminding me just a few years ago AI looked v bleak - seemed like provailing opinion that just one closed source provider would win all.
@bernhardsson @modal I knew it! when I saw Charles started posting about inference a month ago.
I think I have a couple dozen ideas where Modal can curve to make it fit with it's existing infra; will you be up for DMs Erik?
@bernhardsson @modal this is super dope
@bernhardsson @modal Exciting :) congrats Eric, @akshat_b, and team
