DeepSeek V4.1 Flash serving setup: 81 tok/s code on RTX PRO 6000 plus Strix Halo
A home serving stack routes each prompt by predicted time to first token, prefills 8K on an RTX PRO 6000 in about 3.6s, and hits 50-54 tok/s prose and 81 tok/s code with split decode, keeping experts at Q2 quant.
How our DeepSeek V4.1 Flash setup serves a request: one RTX PRO 6000 + two AMD Strix Halo boxes joined by USB4. A router picks the cheapest path for every prompt, by predicted time to first token and time to finish: • Prefix-cache hit: 4–12 ms • Short prompts: prefilled on the two Strix nodes (0.2–1.3 s) • Long prompts: prefilled on the RTX at ~2,150 tok/s (8K in 3.6 s, 32K in 7 s). The state (~2 MB at 2K) is then handed to the nodes in ~0.85 s. Under load the RTX takes anything above ~250 tokens, so the other users’ streams barely pause. Decode: • Nodes alone, with DSpark speculative decoding: 37–39 tok/s prose, 62 code • Split decode, where the RTX runs layers 0–21 and the Strix pair layers 22–39 and only the hidden state crosses each step: 50–54 tok/s prose, 81 code. That’s ×1.33 for one user, with output identical token for token. On quality: for now we’ve chosen to keep the experts at the published Q2 quant rather than push them lower to save memory. We tested alternatives: a 2-bit trellis format scored better on quality but ran ~2× slower on these GPUs, and a lookup-table format matched Q2 but was slower too. So Q2 stays until something beats it on both. The dense layers run a 4-bit trellis format (5-bit head) that measures the same KL against the original model as our 6.5-bit format did, while reading ~20 % fewer bytes per token. Every speedup since then is bit-exact, and the fast RTX prefill path is only used where it passes its quality gate.