DeepSeek V4.1 Flash on 4 DGX Sparks: 3.5-bit EXL3 vs native weights
A developer unpacks Tony's 1.8x chart: the faster bar is a Zig engine running 3.5-bit EXL3 weights, the slower one is vLLM on native 4-bit MXFP4; the second RoCE cable adds only 4% (89.8 to 93.6 tok/s), so the gap comes from engine and quantization, and he stays on native weights to avoid long-reasoning drift.
Hi Andrey! Thanks for pointing this out. Love what @ToNYD2WiLD is doing for the DGX Spark community. He keeps pushing DeepSeek V4.1 Flash on 4 Sparks and shares everything openly: run folders, scripts, evals and honest notes. His "1.8x" chart got me thinking about how tricky it is to compare these setups, so here are a few thoughts. The two bars are two different stacks. The green one is TensorFold, a native Zig engine, running LibertAI's 3.5-bit EXL3 version of the model. The grey one is an earlier vLLM build running the weights as DeepSeek ships them (4-bit MXFP4 experts, FP8 dense layers, BF16 head). Both are impressive bits of engineering. In case anyone wonders about the dual-cable setup: Tony's own numbers show the second RoCE rail adds about 4% (89.8 → 93.6 tok/s). Most of the gap comes from the engine and the smaller weights. Single-user decode on a Spark is mostly limited by memory bandwidth, so fewer bits per weight means faster steps. That's a real trade-off, and it's why I've chosen to stay on native weights for my own stack. The checkpoint is already 4-bit and 8-bit, so going to ~3.5 bits squeezes it a bit further, and I'd rather keep every bit of intelligence DeepSeek shipped. The questions I keep asking myself: will it get stuck in loops, will long reasoning drift, will it hallucinate more? Those are hard to see in short evals (Tony's tool eval is 97.1 vs 98.6 native, very close), and they tend to show up in long reasoning, long code and very long contexts. With that said, there is a new lossless compression technique available thanks to some very smart people and it will be incorporated into my next release. Benchmarking these is honestly hard, and I've tripped over most of these myself: short vs long outputs, measuring after the first token vs end to end, the same prompt on every concurrent stream, code/JSON vs prose (drafts accept a lot more on code), filler vs real text for prefill, and one run vs confidence intervals. Even on the same engine, the text alone moves things about 2x (code 134.8 tok/s vs narrative 67.3 on Tony's chart). I've been experimenting with poisson and goodput but it still isn't there - and my numbers read conservative because of it. It'd be fun to put both stacks on the same ruler, so I'd be glad to run Tony's v1 bench against spark-ds41f. My own release suite uses 256 distinct prompts, 5-minute windows, 95% CIs, real-text prefill to 1M and a temperature-0 determinism gate. Native weights, 1M context, TP4 on 4 Sparks: https://github.com/christopherowen/spark-ds41f Thanks for pushing us all forward, Tony!