DeepSeek V4.1 Flash cuts KV cache to 890 bytes per token

Victor Kuznetsov @wiltodeltaOriginally posted 2026.10.22

Open original on X

DeepSeek at 890 bytes of KV cache proves the agent bottleneck is runtime. In five days, their DeepInfer engine hit 17,780 GitHub stars under Apache-2.0, while DeepSeek-V4.1-Flash-1017 weights shipped under MIT. On the DeepSWE v1.1 coding benchmark, this fast model scores 77.0% against 74.0% for Claude Opus 5 and 73.0% for GPT-5.6 Sol. On Terminal-Bench 4.0 inside the Claude Code harness, it jumped from 31.2% to 42.3%. The weights are only half the story. The real engineering shift happened between memory architecture and runtime. Reusing KV cache across layers in Compressed Sparse Attention 2 with FP4 quantization shrank memory to 890 bytes per token. That is roughly eight times smaller than V4-Flash. On 1M-token agent trajectories with 256K outputs, KV cache size decides how many parallel branches fit on one node. The second detail sits inside the DeepInfer repository. The team open-sourced production kernels for NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs: FlashMLA, DeepGEMM, DeepEP, and DSpark speculative decoding for 5 draft tokens. They also included a deterministic batching flag, DEEPINFER_BATCH_INVARIANT=1, for reproducible debugging. However, they held back online KV-cache cluster management, replica load balancing, and prefill-decode disaggregation across nodes. There is another gift for builders running open models in custom harnesses: the deepseek-recipe package in Rust. In fragile Jinja templates, one stray newline in multi-turn tool history shifts tokenization and degrades accuracy over long runs. Now a strict protocol compiler is locked directly in code. The commoditization boundary has moved. Open weights and local kernels are table stakes, while the production advantage shifts to KV-cache orchestration and exact harness glue. The engine repository link is in the first comment. When your agent pipelines hit a wall on cost or latency, do you swap the model first or rebuild the runtime around it?