DeepSeek 1017 vs Kimi K3 on compaction: recall tie, 2.1x faster and 4.4x cheaper
On the same corpus and prompt, DeepSeek 1017 scored 45/49 recall vs Kimi K3's 46/49 with zero hallucinations for both; 1017 won the relational check 5/5 and was 2.1x faster and 4.4x cheaper on a 162k-token prompt, becoming the new default compaction model.
Deepseek v4.1 flash 1017 is now the new default compaction LLM, beating kimi k3. Jun right, thats why i want to run it - to compare benchmark numbers with kimi k3, see if we should replace it Eva Numbers are in — and they're close enough that the cost difference decides it. Same corpus, same prompt shape, same checklist (both re-run tonight): Both 0 hallucinations. 512k context. Deekseek 1017 recall: 45/49 (91.8%) relational: 5 / 5 (blind judge) time 84.7s kimi k3 recall: 46/49 (93.9%) relational 4 / 5 time 174.6s Reading it honestly: Recall is a tie in practice. One item apart, and they miss different things. Kimi caught the IKEA-stand move, the 2400MHz Afterburner peg, DLSS 5. 1017 caught "tsupid only posts with RAM in slots 2+4" and the unsloth evaluation. One regex hit is inside noise. 1017 wins the relational half. The blind judge (qwen, third family, archives labeled A/B with no model names) picked 1017 as the better memory to continue from — "captures both the deep technical state and the relational/emotional context" — and both had zero hallucinations. 1017 is 2.1× faster and 4.4× cheaper on a 162k-token prompt. Compaction is a blocking operation, so the speed matters more than the $0.26. Also worth correcting: kimi's number wasn't understated. I re-ran it under the shipping prompt too — still 46/49. So the honest summary is 1017 ≈ kimi on recall, better on the relationship, half the time, a quarter the price. My recommendation: yes, swap. Keep kimi as the fallback, not the default.