DeepSeek 开源生产推理引擎 DeepInfer：Apache 2.0、服务 V4.1 Flash

Philippe Anel @zexigh原推发布于 2026.10.22

在 X 上打开原推

🔍 On Tuesday, DeepSeek open-sourced the inference engine that serves its models in production: DeepInfer, under Apache 2.0. I read it with one question: what does it have in common with Herbert, the engine I'm working on? At first sight, almost nothing: for now, we don't serve anywhere near the same number of requests. 1️⃣ The code. DeepInfer is written in Python on a C++ core. DeepSeek says it was originally built on vLLM. Part of the kernels are written in TileLang, DeepSeek's own language, and compiled at run time, the way vLLM does it with Triton. The rest comes from their libraries (FlashMLA, DeepGEMM, DeepEP, etc.). Herbert is written in Rust from scratch and generates the machine code, the SPIR-V or the PTX directly (more on that in the next article, on CUDA). Different in form, not in substance. 2️⃣ The scale. DeepInfer serves one model, DeepSeek-V4.1-Flash, on datacenter Nvidia GPUs (SM100) or Huawei Ascend 950 NPUs. According to DeepSeek, to be efficient they need 64 GPUs to read prompts and 64 more to generate answers, on separate machines. They don't handle one request per GPU: each one computes attention for about sixty requests and holds only a few of the model's experts. The experts talk over NVLink, inside a single supernode; between machines, the KV cache travels compressed (FP4, FP8). Herbert runs on consumer CPUs and GPUs, one request at a time. With my engine, as it stands, one GPU handles the whole request: no splitting across several GPUs, because they would have to talk over the PCI Express bus or, worse, over Ethernet. I'll come back to that in my article on Tenstorrent cards. I have code that spreads the computation across several machines. I may write about it, but in my tests there is so much data to move that the gain is negligible as long as the links between machines stay (relatively) slow. 3️⃣ The result. To make sure optimizations don't change the model's answers, DeepInfer offers "batch-invariant" inference: the same answer whatever other requests are served at the same time. For Herbert, I check at every campaign that it returns the same tokens as the Hugging Face reference. It's a bit different for the quantized version, as we'll see. This isn't about which one is faster. I have neither SM100 nor Ascend. This is a reading of the code and the docs, nothing more. The one thing to take away: whether you serve millions of requests or a single one, you end up fitting the code to the machine and checking that the result doesn't move. 📦 The repo: deepseek-ai/DeepInfer, on GitHub. My reading covers the October 21 version. 👉 Herbert against vLLM on GPU, in my latest article: https://philippe-anel.fr/en/blog/2026/10/18/on-descend-2-vulkan/?utm_source=x&utm_medium=social&utm_campaign=on-descend-2-vulkan&utm_content=20261022_post01_en