DeepSeek 开源 DeepInfer 推理框架，支持 V4.1 Flash

Tech with Mak @techNmak原推发布于 2026.10.22

在 X 上打开原推

DeepSeek just open-sourced the inference engine behind a cluster it says serves 155 trillion tokens a day. And this is becoming quite a pattern. DeepSeek has already released FlashMLA, DeepGEMM, DeepEP, and several other pieces of its inference infrastructure. Now it's publishing the framework that brings much of that engineering together. The project is called DeepInfer, the high-performance inference framework powering DeepSeek's large-scale production services. It currently supports DeepSeek-V4.1-Flash, with several interesting optimizations for running large Mixture-of-Experts models efficiently. The implementation includes expert parallelism with dynamic load balancing, allowing MoE experts to be distributed across GPUs while adjusting for uneven workloads. For decoding, DeepInfer uses DSpark speculative decoding with confidence-scheduled verification. Draft tokens are generated ahead of time, and confidence estimates help determine when verification should happen. The scheduler is also worth studying. CPU-side batch scheduling overlaps with GPU execution, reducing the idle time that normally appears between decoding steps. Other features include sliding-window attention bounded replay, batch-invariant inference, deterministic sampling, and constrained generation using JSON schemas, regular expressions, or EBNF grammars. DeepSeek has also published unusually detailed deployment notes describing how its production system separates prefill and decode, distributes MoE experts across large GPU groups, and reuses cached context. The framework supports NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. One important limitation: the public release currently provides offline inference. The KV cache system, complete prefill/decode disaggregation, and load balancing required for a full online service are not included. DeepInfer was originally built primarily on vLLM, with additional ideas from SGLang and TensorRT-LLM. So this isn't a complete reproduction of DeepSeek's production deployment. But it makes a substantial part of the actual inference implementation available for engineers to inspect, benchmark, and build upon. Fully open source under Apache 2.0. GitHub: https://github.com/deepseek-ai/DeepInfer