DeepSeek open-sources DeepInfer: V4.1-Flash-only serving stack for SM100 and Ascend 950
DeepSeek released DeepInfer, the framework it says serves V4.1-Flash in production, with optimized kernels, speculative decoding, and parallelism setup, but it supports only that model on Nvidia SM100 or Huawei Ascend 950 and omits KV cache, prefill/decode split, and load balancing.
DeepSeek just open-sourced DeepInfer, the framework they say runs their production services for V4.1-Flash. They framed it as handing over the stack so the rest of us can cut compute costs. The repo has the optimized kernels, the speculative decoding bits, and the parallelism setup. It only supports that one model, and only on Nvidia SM100 or Huawei Ascend 950. Older Nvidia cards are out. The README lists what a real serving setup still needs and doesn’t include: a KV cache system, the prefill/decode split, and load balancing. So you get the engine, not the full thing that handles their actual traffic. I’ve spent too many hours on open-source inference drops that sound like the production stack and then leave the expensive parts as an exercise. At least this one says up front what’s missing. If you’ve self-hosted any DeepSeek model, did DeepInfer actually change your setup or are you still fine on vLLM?