deepseek-v4.1IMAGE
DeepSeek Harness vs minimal Pi on the same model: tied quality, 1.63x slower
Same deepseek-flash model on 34 coding/shell/git tasks, two runs each: quality tied at 407/408, but DeepSeek Harness took 1.63x the time and 1.91x the output tokens of minimal Pi.
Zeng Jiajun 曾嘉俊
@zengjiajun_ethArchived 10 days agoOriginally posted 2026.10.17
Zeng Jiajun 曾嘉俊 @zengjiajun_ethOriginally posted 2026.10.17
DeepSeek says V4.1-Flash-1017 was trained for their own harness, so I tested it. Same model (deepseek-flash API, thinking high), same 34 coding/shell/git tasks, 2 runs each, interleaved. Quality tied (407/408 passed). But DeepSeek Harness took 1.63x the time and 1.91x the output tokens vs minimal Pi. Swapping Pi's tool schemas to DSH's: 1.34x slower. Switching Pi to DeepSeek's Messages API: 0.81x, slightly faster. Harness matters, and more harness isn't better.
x.com/zengjiajun_eth/status/2111456956544115165Archived 2026.10.18 16:37