DeepSeek V4.1 Flash reaches 2,100+ tok/s/user on 8× AMD MI355X
Lithos AI co-designs engine, kernels and compiler to hit 2,100+ tok/s/user for DeepSeek V4.1 Flash at full quality on 8× AMD MI355X, 2.8× faster than existing open-source stacks.
Low-latency LLM inference on AMD GPUs requires more than just fast kernels. On MI355X, several architectural challenges introduce significant coordination overhead: * No PDL-equivalent in HIP to overlap dependent kernels. * Distributed L2 caches introduce additional inter-die traffic. * No NVSwitch-style in-network reductions, resulting in up to 4× more inter-GPU communication. We tackle this coordination tax by co-designing inference engine, kernels, and compiler. The result: 2,100+ tokens/s/user for DeepSeek V4.1 Flash at full model quality on 8× AMD MI355X GPUs, 2.8× faster than existing open-source solutions on the same hardware. Excited to share the techniques behind these results, lessons learned from optimizing for MI355X, and how our roofline analysis reveals further opportunities to push inference performance! Tech blog: https://www.lithosai.com/blog/ultra-fast-inference-2100-tokens-per-second-amd-mi355x