DeepSeek announced the open-source release of DeepInfer on Oct. 20. The company says the inference framework powers DeepSeek-V4.1-Flash at scale, with high throughput and low latency. DeepSeek says sharing its production stack is intended to help the community lower compute costs, reduce energy use and make AI more affordable.
What the release includes
The DeepInfer repository describes a release under the Apache License 2.0 license that provides offline inference with core features needed for serving at scale. Its README lists DeepSeek-V4.1-Flash as the supported model, and NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs as supported hardware. It also lists structured outputs and asynchronous scheduling among the framework’s features.
What a complete service still needs
The README lists three requirements not included in this release: a KV cache system, VE/Prefill/Decode disaggregation and load balancing across data-parallel replicas. It says most of the required engine functionality for disaggregation is already implemented, but the released repository does not supply everything needed for a complete inference service.
A separate DeepSeek project, deepseek-recipe, converts API requests into model prompts and model output into responses. Its README says the caller must provide model inference, tool execution and HTTP transport.
DeepInfer’s acknowledgments say the framework was originally built primarily on vLLM. In a quote-post of DeepSeek’s announcement, the vLLM project thanked DeepSeek for the Apache 2.0 release and likewise described DeepInfer as initially built primarily on vLLM.
Comments