SGLang team breaks down M3.1 Flash architecture: MSA, QAT, DSpark
An SGLang-Omni member analyzes the 428B MoE model's sparse attention, quantization-aware training and speculative decoding, citing OSWorld 61.7 and DeepSWE 73.7%.
A few days ago, our SGLang-Omni team experimented with MiniMax M3.1 in a cascaded system. Today, seeing the official release of M3.1 Flash, I spent some time studying its architecture, particularly MiniMax Sparse Attention (MSA). I found the design genuinely interesting! 🤔 M3.1 Flash is a 428B MoE model with only 23B activated parameters per token, supporting up to 1M tokens of context. Beyond MoE, three architectural decisions caught my attention. 1. MiniMax Sparse Attention (MSA) replaces expensive full attention with a lightweight Index Branch that selects relevant KV blocks, followed by a Main Branch performing attention over those blocks. With Top-16 selection and 128-token blocks, the main attention computation can focus on roughly 2K historical tokens instead of the entire context. What I particularly appreciate is that sparsity is incorporated into training through KL alignment and full-attention warmup. The model learns to operate under sparse attention rather than relying on post-training approximations. 2. Quantization-Aware Training (QAT) brings low-precision computation into training for routed experts and attention paths, helping address the substantial memory and bandwidth costs of a 428B MoE model. 3. DSpark Speculative Decoding uses a draft head to propose candidate tokens for parallel verification, reducing the number of expensive sequential decoding steps when acceptance rates are favorable. Meanwhile, the model maintains competitive agentic performance, including 61.7 on OSWorld 2.0 and 73.7% on DeepSWE v1.1. I really like this direction: model architecture and inference efficiency should be co-designed from the beginning, rather than optimized independently after training. Congratulations to the MiniMax team!!!