8×MI355X 上 V4.1 Flash 达 2100+ tok/s/用户

Zhihao Jia @JiaZhihao原推发布于 2026.10.28

在 X 上打开原推

Low-latency LLM inference on AMD GPUs requires more than just fast kernels. On MI355X, several architectural challenges introduce significant coordination overhead: * No PDL-equivalent in HIP to overlap dependent kernels. * Distributed L2 caches introduce additional inter-die traffic. * No NVSwitch-style in-network reductions, resulting in up to 4× more inter-GPU communication. We tackle this coordination tax by co-designing inference engine, kernels, and compiler. The result: 2,100+ tokens/s/user for DeepSeek V4.1 Flash at full model quality on 8× AMD MI355X GPUs, 2.8× faster than existing open-source solutions on the same hardware. Excited to share the techniques behind these results, lessons learned from optimizing for MI355X, and how our roofline analysis reveals further opportunities to push inference performance! Tech blog: https://www.lithosai.com/blog/ultra-fast-inference-2100-tokens-per-second-amd-mi355x