Ant Group and vLLM open-source Humming kernels: 14%–43% throughput gain on V4.1 Flash
Switching V4.1 Flash MoE layers from Marlin W4A16 to Humming W4A8 on eight H20 GPUs raised end-to-end throughput by roughly 14%–43%; ships with vLLM 0.29.0+.
Why does a 4-bit model sometimes run slower than you expected? Often it comes down to the kernels. Ant Group and the vLLM community just open-sourced Humming, one kernel library for quantized inference across many weight and activation formats, dense and MoE, on NVIDIA GPUs from Turing onward. Kernels are compiled on demand from shared CUDA components. In their DeepSeek-V4.1-Flash test on eight H20 GPUs, switching the MoE layers from Marlin W4A16 to Humming W4A8 raised end-to-end throughput by about 14% to 43%. It ships with vLLM 0.29.0 and later. If you serve a quantized MoE model, try the Humming backend on a copy of your real traffic and compare time to first token along with throughput. #PyTorch Link in Comments