DeepSeek discloses context cache scale: 85% hit rate saves ~20k GPUs
V4.1 Flash uses 890 bytes of KV per token, a quarter of V4 Flash; at an 85% hit rate the cache avoids 3.4–4.1x more GPUs, and 100T tokens of context takes 89 PB.
At an 85% hit rate, only about 1/7 input token needs prefill. Without the cache, DS would need 3.4 to 4.1 times as many GPUs for the same traffic, about 20k more. The Q&A's storage server has 20 SSDs of 30.73 TB each, so the whole cache fits on about 183 servers. The GPU number is on the low side, because recomputing a long context costs more per token than recomputing a short one. The storage is cheap because the model's KV is small. V4.1 Flash uses 890 bytes per token, a quarter of what V4 Flash used. With FP8 KV, DeepSeek-V3 uses about 35 KB per token, and Llama 3.1 70B about 160 KB. At 890 bytes, 100T tokens of context take 89 PB. At V3's size, they would take 3.5 EB. DeepSeek has cached context on disk since August 2024. This is the first time they released how big the cache is. The cache system is also one of the parts DS did not disclose. The small KV is part of the model itself, and the weights are open under MIT.